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Abstract 


Semi-Infinite  programming,  that  allows  for  either  infinitely  many  constraints  or  in¬ 
finitely  many  variables  but  not  both,  is  a  natural  extension  of  ordinary  mathematical  pro¬ 
gramming.  There  are  many  practical  as  well  as  theoretical  problems  in  which  the  constraints 
depend  on  time  or  space  and  thus  can  be  formulated  as  semi-infinite  programs.  The  focus 
of  this  dissertation  is  on  formulating  and  solving  semi-infinite  programming  problems.  The 
main  results  includes 

^(1)  An  algorithm  for  solving  a  matrix  rescaling  problem  formulated  as  a  semi-infinite 
linear  program", \Sufficient  conditions  that  guarantee  finite  termination  are  discussed  and 
(^computafionairesults  are  reported. 

(2)  An  algorithm  for  solving  a  matrix  estimation  problem  equivalent  to  a  semi-infinite 
quadratic  program^'  For  a  specified  constant,  this  algorithm  will  find  an  approximate  so- 
/l lution  after  finitely  many  iterations,  or  will  tend  to  an  optimal  solution  in  the  limit.  An 
(  upper  bound  on  the  total  number  of  iterations  needed  for  finding  an  approximate  solution 
is  given.  Computational  results  are  reported. 

- .  (3)  A  one-phase  algorithm  for  solving  a  large  class  of  semi-infinite  linear  programming 

problemsj  yThis  algorithm  has  several  advantages:  it  handles  feasibility  and  optimality 
^'"together  and  can  detect  iiifeasibility  after  a  finite  number  of  iterations;  it  has  very  weak 
restrictions  on  the  constraints;  it  allows  cuts  that  are  not  near  the  most  violated  cut;  and 
it  solves  the  primal  and  the  dual  problems  simultaneously.  Upper  bounds  for  finding  an  e- 
optimal  solution  and  for  the  distance  between  an  e-optimal  solution  and  an  optimal  solution 

are  given.  ^ — - — . 

.  (4)  Applications  of  the  above  algorithm  to  convex  programming. [First,  a  certain  semi¬ 
infinite  linear  program  is  solved  by  this  algorithm  so  as  to  obtain  a  feasible  solution  of  a 
convex  program.  Then,  another  semi- infinite  linear  program  is  solved  by  this  algorithm  so 
as  to  obtain  an  optimal  solution  of  the  convex  program.  In  particular,  it  is  shown  that  for  a 
strongly  consistent  convex  program  this  algorithm  can  find  a  feasible  solution  after  a  finite 
number  of  iterations. 
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Chapter  1 
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Introduction 


1.1  Overview 

Semi-Infinite  programming,  that  allows  for  either  infinitely  many  constraints  or  infinitely 
many  variables  but  not  both,  is  a  natural  extension  of  ordinary  mathemat  ical  programming. 
There  are  many  practical  problems  as  well  as  theoretical  problems  in  which  the  constraints 
depend  on  time  or  space  and  thus  can  be  formulated  as  semi-infinite  programs.  For  example: 

(1)  A  large  class  of  engineering  design  problems  such  as  electronic  circuit  design, 
seismic-resistant  structure  design,  and  single-input  single-output  control  systems  design 
can  be  formulated  as  nonlinear  semi-infinite  programs  (see  Polak  (1981)). 

(2)  The  problem  of  determining  a  cumulative  distribution  function  that  corresponds 
to  a  stochastic  variable  can  be  formulated  as  a  nonlinear  semi-infinite  program  (Gustafson 
and  Kortanek  (1973)). 

(3)  The  educational  testing  problem,  which  concerns  a  reliability  coefficient  (that  mea¬ 
sures  how  reliable  the  students’  total  scores  are  in  an  examination  consisting  of  a  number 
of  subtests),  can  be  formulated  as  a  semi-infinite  linear  program  (Fletcher  (1981)). 

(4)  The  standard  convex  program  can  be  formulated  as  a  semi-infinite  linear  program 
(Dantzig  (1963),  Chapter  24). 

The  focus  of  this  dissertation  is  on  formulating  and  solving  semi-infinite  programming 
problems. 

In  Chapter  2,  we  study  a  matrix  rescaling  problem  that  is  interesting  from  a  purely 
mathematical  point  of  view  as  well  as  having  important  applications.  We  first  formulate  this 


1 


Section  1.1 


Overview 


2 


problem  as  a  semi-infinite  linear  program  and  then  develop  an  algorithm  that  completely 
solves  the  problem  by  taking  advantages  of  its  special  structure.  Sufficient  conditions  that 
guarantee  finite  termination  are  discussed,  and  computational  results  are  reported. 

In  Chapter  3,  we  solve  a  matrix  estimation  problem  that  differs  from  the  ordinary 
least-square  estimation  in  that  the  estimated  matrix  is  required  to  be  positive  definite.  We 
transform  this  problem  into  an  equivalent  semi-infinite  quadratic  program  and  develop  an 
algorithm  to  solve  it.  A  constant  that  measures  the  amount  of  positive  definiteness  required 
of  the  estimated  matrix  must  be  specified.  Given  this  constant,  the  algorithm  will  find  an 
approximate  optimal  solution  after  finitely  many  iterations,  or  will  tend  to  an  optimal 
solution  in  the  limit.  We  also  give  an  upper  bound  on  the  total  number  of  iterations  needed 
for  finding  an  approximate  optimal  solution  and  estimate  how  good  an  approximate  optimal 
solution  is  compared  to  an  optimal  solution. 

In  Chapter  4,  we  present  a  one-phase  algorithm  for  solving  a  large  class  of  semi-infinite 
linear  programming  problems.  This  algorithm  is  based  on  the  algorithms  developed  in 
Chapter  2  and  Chapter  3.  This  algorithm  has  several  advantages:  it  handles  feasibility  and 
optimality  together  and  can  detect  infeasibility  after  a  finite  number  of  iterations;  it  has 
very  weak  restrictions  on  the  constraints;  it  allows  cuts  that  are  not  near  the  most  violated 
cut;  and  it  solves  the  primal  problem  and  the  dual  problem  simultaneously.  We  prove  the 
convergence  of  this  algorithm  in  two  steps.  First,  we  show  that  this  algorithm  can  find  an 
e-optimal  solution  after  finitely  many  iterations.  Then  we  use  this  result  to  show  that  it  can 
find  an  optimal  solution  in  the  limit.  We  also  estimate  how  good  an  e-optimal  solution  is 
compared  to  an  optimal  solution  and  give  an  upper  bound  on  the  total  number  of  iterations 
needed  for  finding  an  e-optimal  solution  under  some  assumptions. 

In  Chapter  5,  we  show  how  to  solve  a  convex  program  by  semi-infinite  linear  program¬ 
ming.  We  apply  the  general  algorithm  presented  in  Chapter  4  to  solve  a  certain  semi-infinite 
program  so  as  to  obtain  a  feasible  solution  of  the  convex  program,  and  then  use  this  solution 
as  a  starting  point  of  another  semi-infinite  program  to  obtain  an  optimal  solution  of  the 
convex  program.  In  particular,  for  a  strongly  consistent  convex  program  the  algorithm  can 
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find  a  feasible  solution  after  finitely  many  iterations. 

1.2  Preliminaries 

The  primal  problem  of  semi-infinite  programming  is  defined  as: 

(SIP)  minimize  f(z) 

subject  to 

g(x,u )  >  0  for  all  u  £  U, 

where  x  £  Rn ,  f  :  Rn  — >  R' ,  U  C  Rm  is  an  infinite  parameter  set,  and  g  :  Rn  x  U  — ►  R' . 

An  important  class  of  semi-infinite  programming  problems  are  semi-infinite  linear  pro¬ 
grams  in  which  the  objective  function  is  linear  and  the  constraints  are  linear  in  x  for  any 
fixed  u£U.  Explicitly,  the  primal  problem  of  semi-infinite  linear  programming  is: 

(SIL)  minimize  cTx 

subject  to 

a(u)T x  —  6(u)  >  0  for  all  u  £  U, 

where  c,  x  £  Rn ,  U  is  an  infinite  parameter  set,  a  :  U  — +  Rn,  and  b  :  U  —+  R*  ■  The  dual 

program  of  (SIL)  is  defined  as  an  infinite  collection  of  finite  dual  linear  programs:  for  all 

finite  subsets  {«'  :  i  £  A}  of  U 

(SID)  maximize 
subject  to 

a(«‘)y.  =  c 

A  is  finite  and  {u'  :  i  £  A}  C  U 
y,  >  0  for  all  *  £  A  . 

Semi-infinite  programming  has  been  studied  since  the  early  1960s.  So  far  there  are 
three  types  of  methods  for  solving  semi-infinite  programs:  discretization-type  methods, 
semi-continuous-type  methods,  and  continuous-type  methods  (see,  e  g.,  Hettich  (1979)). 

A  discretization- type  method  generates  U ,  a  finite  subset  of  the  infinite  parameter 
set  U,  and  finds  an  x  that  minimizes  f(x)  over  finitely  many  constraints  g{x,u)  >  0  for 
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all  u  €  U.  If  the  grid  size  d  =  max.gr/'  minfi€f>||u  —  u||  is  sufficiently  small,  then  x  is 
an  approximate  optimal  solution  to  (SIP).  The  discretization-type  methods  are  limited  to 
those  U  having  nice  properties,  e.g.,  U  =  {ti  €  ft1  :  0  <  u  <  1},  and  generally  fail  when  U 
is  unbounded.  The  disadvantage  of  this  type  of  method  is  that  sometimes  the  approximate 
solution  x  may  not  be  satisfactory. 

A  semi-continuous-type  method  generates  a  sequence  of  finite  subsets  of  U ,  {(/*  :  Jfc  = 
1,2, ... }  with  Uk  C  Uk+ 1  for  all  k,  and  finds  xk  =  argmin{/(i)  :  g(x,  u)  >  0  for  all  u  € 
{/*},  where  {Uk  :  Jfc  =  1, 2, . . . }  are  generated  in  such  a  way  that  any  cluster  point  of  the 
sequence  (xfc  :  Jfc  =  1,  2, . . .  }  is  an  optimal  solution  of  (SIP).  Semi-continuous-type  methods 
differ  in  the  ways  they  generate  {Uk  :  k  =  1,2, ...  },  some  being  more  efficient  than  others 
for  the  problem  at  hand. 

A  continuous-type  method  typically  finds  a  finite  set  of  differentiable  convex  functions 
hi(x),  i  =  1,...,  t  such  that  {x  :  /»*(*)  <  0, i  =  1,...,<}  is  the  same  as  (i  :  g(x,u)  > 
0  for  all  u  €  U}.  When  this  can  be  done,  the  semi-infinite  program  can  be  reduced  to 
a  standard  convex  program:  minx{/(ar)  :  hi(x)  <  0,  i  =  1, . . . ,  <}.  The  drawback  of 
continuous-type  methods  is  that  the  assumptions  are  often  restrictive  and  convergence  is 
often  only  local. 


Chapter  2 


Rescaling  a  Matrix  Positive  Definite 


2.1  Introduction 

We  solve  the  following  matrix  rescaling  problem: 

Given  a  square  real  matrix  A/,  does  there  exist  a  positive  diagonal  matrix  D  such  that  DM 
is  positive  definite?  If  such  a  D  exists,  how  can  it  be  constructed? 

Such  questions  arise  in  mathematical  economics  and  in  the  study  of  certain  engineering 
systems  (see,  e.g.,  Arrow  and  McManus  (1958),  and  Araki  (1975))-  A  necessary  and  suffi¬ 
cient  condition  for  the  existence  of  such  a  D  for  a  3  x  3  matrix  was  given  by  Cross  (1978). 
A  necessary  and  sufficient  condition  for  the  existence  of  D  for  the  general  n  x  n  case  was 
given  by  Barker,  Berman  and  Plemmons  (1978).  However,  their  condition  is  difficult  to 
verify  in  practice.  Methods  for  constructing  such  D's  for  special  classes  of  matrices  were 
discussed  in  Barker,  Berman  and  Plemmons  (1978),  and  Berman  and  Hershkowitz  (1983). 
The  existence  of  D  for  a  square  Leontief  (Minkowski)  matrix  was  proved  by  Tartar  (1971) 
and  Dantzig  (1983).  The  first  algorithm  for  solving  the  general  problem  was  given  by  Khalil 
(1982).  He  defines  a  function  g(x)  to  be  the  smallest  eigenvalue  of  D(x)M  +  Mr D(x),  where 
D(x)  is  a  diagonal  matrix  with  diagonal  elements  xj,  . .. ,  x„.  His  idea  is  that  D(x)M  is 
positive  definite  if  and  only  if  g(x)  is  positive.  His  algorithm  finds  a  positive  x  such  that 
g(x)  >  0  as  follows:  At  iteration  k ,  if  g{xk)  >  0,  then  stop  because  D(xk)M  is  positive 
definite;  otherwise,  find  a  direction  dk  by  solving  a  linear  program  and  then  construct  x*+1 
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as  a  convex  combination  of  xk  and  dk  such  that  rt+1  is  in  the  interior  of  the  unit  box  of 
Rn.  If  such  D's  exist,  then  his  algorithm  will  find  one  after  a  finite  number  of  steps.  If  his 
algorithm  goes  on  infinitely,  then  there  is  no  positive  diagonal  matrix  D  such  that  DM  is 
positive  definite. 

Herein  we  present  an  alternative  way  of  solving  this  problem.  Our  approach  is:  first 
formulate  the  matrix  rescaling  problem  as  a  semi-infinite  linear  program  and  then  solve 
the  semi-infinite  linear  program  by  generating  and  solving  a  sequence  of  standard  linear 
programs.  Our  algorithm  is  “almost”  finite.  To  see  this,  let  us  divide  square  matrices  M 
into  three  classes. 

Class  1.  M  that  can  be  rescaled  positive  definite  (i.e.,  DM  is  positive  definite  for  some 
positive  diagonal  matrix  D). 

Class  2.  M  that  cannot  be  rescaled  positive  definite  and  moreover  cannot  be  rescaled 
positive  semidefinite  (i.e.,  DM  is  not  positive  semidefmite  for  any  nonnega-  tive  nonzero 
diagonal  matrix  D). 

Class  3.  M  that  cannot  be  rescaled  positive  definite  but  can  be  rescaled  positive 
semidefinite  (i.e.,  DM  is  positive  semidefinite  for  some  nonnegative  nonzero  diagonal  matrix 
D). 

Given  any  Class  1  matrix  A/,  our  algorithm  can  find  a  positive  diagonal  matrix  D  such 
that  DM  is  positive  definite  after  a  finite  number  of  steps.  Given  any  Class  2  matrix  A/, 
our  algorithm  can  detect  that  M  cannot  be  rescaled  positive  definite  after  a  finite  number 
of  steps.  Given  any  Class  3  matrix  A/,  our  algorithm  can  detect  that  M  cannot  be  rescaled 
positive  definite  in  the  limit.  But,  a  matrix  M  is  of  class  3  if  and  only  if  the  origin  is  on 
the  boundary  of  a  convex  set  (see  Lemmas  2.2  and  2.3),  or  equivalently,  if  and  only  if  a 
particular  function  of  the  elements  of  M  is  zero  (see  Remark  2.3),  which  can  only  happen 
for  a  set  of  very  special  M  of  measure  zero.  Thus  our  algorithm  is  “almost”  finite.  Another 
advantage  of  our  algorithm  is  that  it  is  not  compromised  by  small  errors  in  the  calculation 
of  eigenvalues  and  eigenvectors. 

The  content  of  this  chapter  is  as  follows:  in  Section  2.2  we  explain  notation  and  prelim- 
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inaries.  In  Section  2.3  we  formulate  the  problem  as  a  semi-infinite  linear  program,  specify 
the  algorithm,  prove  its  correctness  and  convergence,  give  conditions  that  guarantee  finite 
termination,  and  finally  present  necessary  and  sufficient  conditions  for  M  to  be  rescaled 
positive  definite.  In  Section  2.4  we  report  on  computational  results.  Finally,  we  discuss 
possible  ways  to  accelerate  the  convergence  in  Section  2.5. 

2.2  Notation  and  Preliminaries 

We  define  an  n  x  n  real  matrix  M,  not  necessarily  symmetric,  to  be  positive  definite  if 
xT  Mx  >  0  for  all  0  ^  x  €  Rn,  and  to  be  positive  semidefinite  if  xrMx  >  0  for  all  x  £  Rn . 

If  there  exists  a  positive  diagonal  matrix  D  such  that  DAI  is  positive  definite,  we  say 
that  M  can  be  rescaled  positive  definite.  Such  matrices  are  called  “diagonally  stable”  in 
Barker,  Berman  and  Plemmons  (1978).  “Lyapunov  diagonally  stable”  in  Hershkowitz  and 
Schneider  (1985),  and  “Volterra-Lyapunov  stable”  in  Cross  (1978). 

Superscripts  on  vectors  are  used  to  denote  different  vectors,  while  subscripts  are  used 
to  denote  different  components  of  a  vector. 

Let  S"-1  =  {x  £  Rn  ■  xT x  =  1}  denote  the  unit  sphere  in  Rn  and  ={x£  Sn~ 1  : 
x  >  0}  denote  the  set  of  nonnegative  vectors  in  5n~* . 

D(x)  is  a  diagonal  matrix  with  diagonal  elements  x,  for  i  =  1,  .  . . ,  n. 

For  a  real  symmetric  matrix  B ,  let  \{B]  stand  for  the  smallest  eigenvalue  of  B  and 
V[B]  a  corresponding  eigenvector  of  unit  length. 

Given  a  mathematical  programming  problem  (P),  r(P)  denotes  the  optimal  objective 
function  value  of  (P). 

Let  e'  be  the  i-th  unit  vector  of  R”  and  e  =  c1  •  •  •  +  en. 

||x||  denotes  the  Euclidean  norm  of  x. 

Fact  2.1.  M  is  positive  definite  if  and  only  if  Af  4-  AfT  is  positive  definite. 

Fact  2.2.  Af  is  positive  definite  if  and  only  if  xT Alx  >  0  for  all  x  £  Sn~' . 

Fact  2.3.  All  principal  minors  of  Af  remain  sign  invariant  under  a  positive  rescaling 
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Fact  2.4.  If  M  is  positive  definite,  then  a  11  its  principal  minors  are  positive  (see,  e.g., 
Cottle  (1964)). 

Fact  2.5.  If  M  is  positive  definite,  then  BT MB  is  positive  definite  for  any  real 
nonsingular  matrix  B . 

Fact  2.6.  For  any  real  symmetric  matrix  B,  A[B]  =  min{uTSu  :  u  £  Sn_1 }  (see,  e.g., 
Wilkinson  (1965),  pp.98-99). 

Fact  2.7.  For  any  real  symmetric  matrix  B,  A[B]  is  a  continuous  function  of  the 
elements  of  B  (see,  e.g.,  Isaacson  and  Keller  (1966),  p.  136). 

Remark  2.1. 

(1)  Let  DM  (M  D)  be  a  positive  rescaling  of  the  rows  (columns)  of  the  matrix  M . 
Then,  M  can  be  column-rescaled  positive  definite  if  and  only  if  M  can  be  row-rescaled 
positive  definite.  Indeed,  if  DM  is  positive  definite,  where  D  is  a  positive  diagonal  matrix, 
then  ( D~')T DM D~ 1  =  MD"1  is  also  positive  definite  (Fact  2.5)  and  vice  versa. 

(2)  We  are  only  interested  in  rescaling  nonsymmetric  matrices  because  if  a  real  sym¬ 
metric  matrix  is  not  positive  definite,  then  it  cannot  be  rescaled  positive  definite.  Indeed,  if 
M  can  be  rescaled  positive  definite,  then  (by  Facts  2.3  and  2.4)  all  its  principal  minors  are 
positive.  If  a  real  symmetric  matrix  is  not  positive  definite,  then  at  least  one  of  its  leading 
principal  minors  is  not  positive  and  thus  it  cannot  be  rescaled  positive  definite. 

(3)  If  M  can  be  rescaled  positive  definite,  then  it  is  easy  to  see  that  M  is  nonsingular 
and  the  diagonal  elements  of  M  are  positive  (by  Fact  2.4,  d,m,,  >  0  for  all  i,  which  implies 
mu  >  0  for  all  i).  Therefore,  without  loss  of  generality  we  assume  that  the  matrix  M  to  be 
rescaled  is  nonsymmetric,  nonsingular  and  has  only  positive  diagonal  elements. 

2.3  An  Algorithm  and  its  Convergence 

First  we  show  that  solving  the  matrix  rescaling  problem  is  equivalent  to  finding  a  solution 
of  an  infinite  system  of  linear  inequalities. 


Theorem  2.1.  Suppose  that  all  diagonal  elements  of  the  matrix  M  are  positive.  Then  M 
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can  be  rescaled  positive  definite  if  and  only  if  the  infinite  system  of  linear  inequalities 

(ISLI)  :  [D(u)Mu]Tx  >  1  for  all  u  G  Sn~' 

has  a  solution.  Moreover,  if  x  is  any  solution  of  (ISLI),  then  D(x)  rescales  M  positive 
definite. 

Proof.  We  will  make  frequent  use  of  the  identity  D(u)x  =  D(x)u  for  all  u  and  x.  If  there 
exists  a  positive  diagonal  matrix  D(d)  such  that  D(d)M  is  positive  definite,  let  /(u)  = 
ur D(d)Mu  for  u  €  Sn_1 .  Since  /(u)  is  a  continuous  function  of  u  and  Sn_1  is  a  compact 
set,  /(ti)  achieves  its  infimum  on  Sn_1 ,  i.e.,  there  exists  ti  €  5”1-1  such  that  f(u)  >  f(u)  >  0 
for  all  u  €  Sn_1.  Let  x  =  d//(u),  then 

[D(u)Mu]tx  =  [D(u)Mu]rd/f(u) 

=  uTMTD(u)d/f(u) 

=  ur  Mr  D(d)u/  f(u) 

=  [urD(d)Mu]r/f(u) 

=  /(«)//(«) 

>  1 

for  all  u  6  5n_1.  Thus  (ISLI)  has  a  solution.  On  the  other  hand,  if  x  is  a  solution  of  (ISLI), 
then  because  ur D(x)Mu  is  a  scalar,  we  have 

urD(x)Mu  =  [uT  D(x)Mu]r 
=  uT  MT  D(x)u 
=  t irMrD(u)x 
=  [D(u)Mu]Tx 
>  1 

for  all  u  6  S’"-1.  By  Fact  2.2,  D{x)M  is  positive  definite.  To  complete  the  proof,  we  only 
need  to  show  that  any  feasible  x,  in  particular  x ,  is  positive.  Let  e*  be  the  unit  vector  with 
1  in  component  i.  Choose  u  =  e'  €  Sn-\  then  x ,  =  {e')T D(x)Me'/mi,  >  1  /mtl  >  0.  | 
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Theorem  2.1  tells  us  that  M  can  be  rescaled  positive  definite  if  and  only  if  (ISLI)  has 
a  solution;  and,  moreover,  for  any  z  solving  (ISLI),  D(z)  rescales  M  positive  definite.  The 
algorithm  we  are  going  to  present  is  designed  to  test  whether  (ISLI)  has  a  solution  or  not 
and  to  find  such  a  solution  if  it  exists. 

It  is  well  known  that  deciding  whether  a  finite  system  of  linear  inequalities  has  a 
solution  is  equivalent  to  solving  a  linear  program  (see,  e  g.,  Dantzig  (1963),  Chapter  5). 
In  an  analogous  way,  in  order  to  solve  the  infinite  system  of  linear  inequalities  (ISLI),  we 
convert  it  into  a  semi-infinite  linear  program  (SILP)  by  assigning  to  it  an  objective  function 
to  be  minimized  such  as  below: 

(SILP)  minimize  eTz 
subject  to 

[D(u)Mu]rz  >  1  for  all  u  £  Sn~\ 

where  eT  =  (1, 1, . . .  ,  1)  is  a  row  vector  with  n  ones.  The  dual  of  a  semi-infinite  linear 
program  (DILP)  was  defined  earlier  in  Section  1.2  as  an  infinite  collection  of  finite  dual 
linear  programs:  For  all  finite  subsets  {u*  :  i  €  A}  of  5n_1 , 

(DILP)  maximize  yt 

subject  to 

E,€a  (0(u*)Mu')y,  =  e 

y,  >  0  for  all  i  £  A 

A  is  finite  and  {u*  :  i  £  A}  C  Sn_1 . 

The  algorithm  generates  and  solves  a  sequence  of  linear  programs  LP(fc),  the  restriction 
of  (DILP)  to  n  +  fc  columns.  We  will  show  that  if  the  value  of  the  objective,  t>(LP(&)),  tends 
to  infinity,  then  M  cannot  be  rescaled  positive  definite  (Theorem  2.3).  Otherwise,  we  will 
show  that  a  positive  vector  x  which  rescales  M  positive  definite  can  be  found  after  finitely 
many  iterations  (Theorem  2.2). 

As  noted  earlier,  we  assume  that  the  input  matrix  M  is  nonsingular  and  has  only 
positive  diagonal  elements  (Remark  2.1). 
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Algorithm  2.1. 

Step  1  (Initialization). 

Let  k  :=  0; 

let  «  <  1  be  a  small  positive  number  (e.g.,  e  =  I0-fi); 
let  LP(0)  be  the  linear  program 

maximize  y, 

subject  to 

ELi  (£>(e*)Me*)jfc  =  e 

yi  >  0  for  all  i  =  1, . . . ,  n. 

Step  2. 

Let  **  be  an  optimal  dual  solution  of  LP(k). 

Find  a  A*  satisfying  |Afc  —  \[D(xk)M  -f  MrD(xk)]\  <  (l/2)e. 

If  A*  >  (l/2)e,  then  D{xk)M  is  positive  definite,  stop. 

Step  3. 

Find  a  vector  u*+1  such  that 

||ufc+i  _  V[D(xk)M  +  MT D(xk)]\\  <  e  and  (ufc+,)rD(xfc)Mut+1  <  e. 

Form  LP(fc  +  1)  by  adjoining  the  column, 

to  the  constraint  matrix  of  LP(fc)  with  “cost”  coefficient  =  1; 

solve  LP(fc  +  1). 

If  v(LP(l  +  1))  =  oo,  then  M  cannot  be  rescaled  positive  definite,  stop. 

Else,  k  :=  k  +  1,  go  to  Step  2. 

Comments  on  Algorithm  2.1. 

(1)  Since  we  assume  that  >  0  for  all  i  =  1,  ... ,  n,  LP(0)  is  feasible.  Therefore, 
LP(fc)  is  feasible  for  all  k. 

(2)  Efficient  algorithms  for  calculating  approximate  eigenvalues  and  eigenvectors  of  a 
matrix  are  discussed  in  Wilkinson  (1965).  The  amount  of  computational  effort  to  compute 
these  approximations  depends  on  the  error  bound  c. 
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(3)  A  symmetric  matrix  B  is  positive  definite  if  and  only  if  A[B]  >  0;  in  general  this 
is  not  true  for  nonsymmetric  matrices.  Therefore,  we  calculate  the  smallest  eigenvalue  of 
the  symmetric  matrix  D(xk)M  +  MrD(xk)  in  order  to  know  whether  D(xk)M  is  positive 
definite  or  not  (Fact  2.1). 

(4)  If  the  algorithm  does  not  stop  at  a  certain  iteration  k,  then  it  generates  u*+1 
satisfying  [£>(ufc+1  )A/ufc+,]rx*  <  e  <  1.  While  [D(u*)M u']Tx*  >  1  for  all  i  =  1,  . . . ,  Jfc 
since  xk  is  an  optimal  dual  solution  of  LP(fc).  Therefore,  no  column  D(uk)Muk  can  be 
brought  in  more  than  once. 

Next,  we  prove  the  correctness  and  convergence  of  the  algorithm  and  discuss  conditions 
that  guarantee  termination  in  a  finite  number  of  iterations. 

Theorem  2.2.  If  there  exists  a  positive  diagonal  matrix  D  such  that  DM  is  positive 
definite,  then  the  algorithm  can  find  such  a  D  after  finitely  many  iterations. 

Proof.  If  M  can  be  rescaled  positive  definite,  then  the  corresponding  program  (SILP) 
is  feasible  by  Theorem  2.1.  Let  x  be  a  feasible  solution  of  (SILP).  Then,  for  all  feasible 
solutions  of  (DILP),  we  have 

1-j  y'  -  ]C  y«[£)(u‘)Mti,]rx  =  eTx. 

«€A  t€A 

Namely,  the  objective  function  of  (DILP)  is  bounded  from  above  by  erx.  The  algorithm 
generates  a  sequence  of  linear  programs  LP(ib).  Each  LP(ifc)  is  feasible  and  is  the  restriction 
of  (DILP)  to  certain  columns.  Hence,  the  objective  function  of  LP(fc)  is  also  bounded  from 
above  by  eTx.  Recall  that  the  first  n  columns  of  the  constraint  matrix  of  LP(ib)  are  m^e* 
for  i  =  1,  ...,  n  and  therefore  any  dual  solution  x  satisfies  m,,x,  >  1,  m,,  >  0.  By  the 
duality  theorem  of  linear  programming,  we  can  find  an  optimal  dual  solution  xk  of  LP(Jb) 
which  satisfies 

0  <  ( mn)~x  <  x*  <  erxk  <  eTi  for  all  i  =  1, . . . ,  n. 


Let  T  =  {x  6  Rn  ■  0  <  Xj  <  erx  for  all  t  =  l,...,n}  and  F(x,u)  =  ( D(u)Mu]Tx .  Then 
F(x,u)  is  uniformly  continuous  on  T  x  5n_1,  i.e. ,  for  any  b  >  0,  there  exists  q  >  0  such 
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(2.1) 


that 

||(*,  ti)  -  (*,  ti)||  <  TJ  implies  | F(x,  u)  -  F(x,  u)|  <  6 

for  all  (x,  u)  and  (x,  u)  €  T  x  S'*-1 . 

In  particular,  for  6  =  1  —  e  >  0,  there  exists  rj  >  0  such  that  (2.1)  holds.  If,  on  the  contrary, 
we  were  to  assume  that  the  algorithm  goes  on  infinitely,  then  it  generates  uk  6  Sn~'  for 

k  =  0,  1, _ Because  Sn_1  is  compact,  for  the  rj  >  0,  there  exists  u*  and  uJ  in  the  sequence 

satisfying  ||u‘  —  uJ||  <  fj.  Without  loss  of  generality,  we  assume  that  i  <  j.  Because  the 
algorithm  does  not  stop  at  iteration  j  —  1  and  i  <  j,  we  have 


F(xj~\u')  =  [D(u*)Mu‘]7V-1  >  1, 


(2.2) 


and 


F(x>-\ u>)  =  [D(uJ)Mu-»]r*J_1  <  e.  (2.3) 

However,  (2.2)  and  (2.3)  imply  that 

II^-1 ,  u’)  -  F(x>~' ,  uJ )|  >  1  -  c  =  l  while  IK*'-1 ,  «*)  -  (*>"' ,  u^)||  <  t), 

which  contradicts  the  uniform  continuity  of  F(x,u)  on  T  x  Sn_1.  It  follows  that  the 
algorithm  must  be  finite  for  matrices  which  can  be  rescaled  positive  definite.  | 


Remark  2.2.  We  have  proved  the  finiteness  of  the  algorithm  under  the  assumption  that 
M  can  be  rescaled  positive  definite.  In  fact,  the  boundedness  of  v(LP(t))  =  eTxk  for  k  =  1, 
2,  . . .  is  the  only  assumption  we  need  for  the  proof.  Since  the  feasible  region  of  LP(fc)  is 
contained  in  the  feasible  region  of  LP(ib+  1),  i/(LP (£  +  1))  >  v(LP(ife))  for  all  k.  Therefore, 
in  the  case  M  cannot  be  rescaled  positive  definite,  the  algorithm  generates  a  sequence  of 
feasible  solutions  of  (DILP)  whose  objective  function  values  tend  increasingly  to  infinity. 
Hence: 


Theorem  2.3.  The  following  are  equivalent: 
(1)  M  cannot  be  rescaled  positive  definite; 
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(2)  v(DILP)  =  oo; 

(3)  /imt_00t/(LP(i))  =  oo.  | 

We  have  seen  that  the  algorithm  is  finite  in  the  case  M  can  be  rescaled  positive  definite. 
We  now  give  a  condition  which  ensures  the  finiteness  of  the  algorithm  in  the  case  M  cannot 
be  rescaled  positive  definite,  i.e.,  there  exists  a  finite  j  such  that  v(LP(j))  =  oo. 

Condition  2.1.  M  cannot  be  rescaled  positive  definite  if  for  every  nonnegative  and  nonzero 
diagonal  matrix  D,  there  exists  a  u  €  S’"-1  such  that  uT DMu  <  0.  Equivalently,  if  there 
does  not  exist  a  nonnegative  and  nonzero  diagonal  matrix  D  such  that  DM  is  positive 
semidefinite  or  positive  definite. 


Theorem  2.4.  If  M  satisfies  Condition  2.1,  then  the  algorithm  terminates  after  finitely 
many  iterations. 


Proof.  Define  G(x)  =  A[(D(x)Af  +  MTD{x))/ 2]  =  A[D(x)M  +  MrD(x)]/2.  It  follows 
from  Fact  2.7  that  G(x)  is  a  continuous  function  of  x.  Since  M  satisfies  Condition  2.1,  we 
have  (Fact  2.6) 

G(x)  =  minimum  uT  D(x)Mu  <  0  for  all  x  6  S"-1 , 
where  S’"-1  =  {x  6  S'*-1  :  x  >  0}.  Therefore, 


0  =  maximum  G(x)  =  maximum  minimum  ur Mu  <  0. 

Let  F(x,u )  =  [D{u)M\i]T x  —  urMrD(x)u  =  [D(x)Mu]ru.  Let  6  =  —0/2  >  0,  then  there 
exists  T]  >  0  such  that 

1 1 (x ,  u)  -  (x,  u)||  <  T}  implies  | F(x,  u )  -  F(x ,  u)|  <  6  =  -0/2 

(2.4) 


for  all  (x,  u)  and  (x,  u)  G  5"  1  x  5”  1 . 

As  indicated  in  the  proof  of  Theorem  2.2,  if  we  assume  on  the  contrary  that  the  algorithm 
goes  on  infinitely,  it  generates  u*  and  u1  (i  <  j )  satisfying  1 1 ti *  —  u J 1 1  <  T)  and  F(xJ~'  ,u')  >  1. 
Let  x-»_1  =  xJ-1/||x;-1!|.  Since  F(x,u)  is  homogeneous  of  degree  one  in  x, 


F(xJ_1 ,  u')  >  l/||xJ—1 1|  >  0. 


(2.5) 
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Let  w  =  V[D(iJ-1)Af  +  MrD(*^_1)J.  Since  w  is  normalized,  w  =  V[(D(z>~l)M  + 
MT D(x]~'  ))/2].  Now  choose  the  e  of  Step  1  of  the  algorithm  such  that  e  <  rj.  Then, 
||  tiJ  —  u>||  <  c  <  ij  in  Step  3  and  hence  we  have 

I  F(&-' ,  t i>)~  F(z>~'  ,w)\<  6  =  -0/2 

by  the  uniform  continuity  of  F(z,u)  on  S+-1  x  S"-1 .  Thus, 

F{z>~' ,  uj)  <  -0/2  +  wTD(zj~'  )Mw 

=  -0/2  +  ( \/2)wt[D{z>~')M  +  MrD(z>-')]tv 
=  -0/2  +  A  [{D(z}~')M  +  A/tD(*j~1))/2] 

(2.6) 

=  -0/2  +  G(z>~') 

<-0/2  +  0 

=  0/2. 

However,  (2.5)  and  (2.6)  imply  that  |f,(i>~1,  u‘)  -  u-j)|  >  — (1/2)  ^  =  6  while 

||(xJ-1,u*)  —  (zJ~\uJ)  ||  =  ||  u*  -  uJ  ||  <  rf,  which  contradicts  the  uniform  continuity  of 
F(z,  u)  on  5“-1  x  5n-1 .  Therefore,  if  M  satisfies  Condition  2.1,  then  the  algorithm  termi¬ 
nates  after  finitely  many  iterations.  | 

In  the  rest  of  this  section,  we  discuss  other  necessary  and  sufficient  conditions  for  M 
to  be  rescaled  positive  definite  and  equivalent  statements  of  Condition  2.1.  First,  we  state 
a  theorem  that  we  are  going  to  use. 

Alternative  Theorem.  Let  hi(t)  for  i  =  1,  . . . ,  n  be  n  real-valued  functions  of  t  £  T 
where  T  may  be  finite  or  infinite.  UU=  {(/»,(! ), . . . ,  hn(t))  :t  £T)  is  closed,  then 

(1)  The  system  ... ,  hn(t))Tz  >  0  for  all  t  £  T  has  a  solution  if  and  only  if  the 

origin  is  not  contained  in  conv(U),  the  convex  hull  ofU. 

(2)  The  system  (hi(t),  ....  hn(t))Tz  >  0  for  all  t  £  T  has  a  non-trivial  (i.e.,  nonzero ) 
solution  if  and  only  if  the  origin  is  not  contained  in  the  interior  of  conv(U)  (Dines  and 
McCoy  (1933)).  | 
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Lemma  2.1.  The  matrix  M  with  a  positive  diagonal  cannot  be  rescaled  positive  definite 
if  and  only  if  {x  :  [£>(«)A/u]rar  >  0  for  all  u  £  Sn_1 }  is  empty. 

Proof.  Let  P  =  {z  :  [Z?(u)Afujrx  >  0  for  all  u  £  S’"-1 }.  If  P  is  empty,  then  (ISLI)  has  no 
solution.  By  Theorem  2.1,  M  can  not  be  rescaled  positive  definite.  On  the  other  hand,  if 
P  is  not  empty,  let  x  £  P.  Then,  urD(x)Mu  =  [D(ti)Mu]rx  >  0  for  all  u  £  Sn_1 .  Hence, 
D(x)M  is  positive  definite  and  x  is  a  positive  vector.  | 

Lemma  2.2.  M  cannot  be  rescaled  positive  definite  if  and  only  if  the  origin  is  contained 
in  conv(D(v)Afu  :  u  £  Sn~t). 

Proof.  It  is  easy  to  show  that  (D(u)Mu  :  u  £  S'*'1}  is  closed.  The  lemma  then  follows 
easily  from  the  Alternative  Theorem  and  Lemma  2.1.  | 

Condition  2.2.  0  €  int{conv[D(u)Mti  :  ti  £  S’1""1]}  where  int{5}  denotes  the  interior  of 

a  set  S- 

Lemma  2.3.  Condition  2.1  and  Condition  2.2  are  equivalent. 

Proof.  If  M  satisfies  Condition  2.2,  then  the  system 

urD(z)Mu  =  [D(u)Mu]rz  >  0  for  all  u  £  Sn_1 

has  no  non-trivial  solutions  (by  the  Alternative  Theorem).  This  implies  for  any  0  ^  d  >  0, 
there  exists  u  e  5n-'  such  that  ur D(d)Mu  <  0,  i.e.,  Af  satisfies  Condition  2.1.  On  the 
other  hand,  if  M  does  not  satisfy  Condition  2.2,  then  (by  the  Alternative  Theorem)  there 
exists  x  0  such  that  uTD(x)Mu  =  [D(u)Af u]rx  >  0  for  all  u  £  5n_1 .  In  particular, 
(e')r£>(x)Afe*  =  Xjm.j  >  0  for  all  i  =  1,  ...,  n.  This  implies  that  Xj  >  0  for  all  i  =  1, 

. . . ,  n.  (we  are  assuming  that  all  diagonal  elements  of  M  are  positive).  Hence,  M  does  not 
satisfy  Condition  2.1.  | 

Remark  2.3.  Notice  that  G(z)  =  minimum  ].e,<;n-iurD(x)Afu  is  a  continuous  function 
of  x.  It  is  not  hard  to  show  the  following: 
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(1)  M  can  be  rescaled  positive  definite  if  and  only  if 

maximum  minimum  uT Mu  >  0. 

x€Sn-1  «€■*>*“’ 

(2)  Condition  2.1  or  2.2  is  equivalent  to 

Condition  2.3.  maximum  minimum uTMu  <  0. 

We  now  summarize  all  necessary  and  sufficient  conditions  for  M  to  be  rescaled  positive 
definite  in  Theorem  2.5. 

Theorem  2.5.  If  all  diagonal  elements  of  M  are  positive,  then  the  following  are  equivalent: 

(1)  M  can  be  rescaled  positive  definite ; 

(2)  (ISLI):  [D(u)Mu]r x  >  1  for  all  u  €  Sn_1  has  a  solution; 

(3)  [D(u)Mu]r  x  >  0  for  all  u  £  S’*-1  has  a  solution; 

(4)  the  origin  is  not  contained  in  conv(D(u)Mu  :  u  £  Sn_1 ); 

(5)  maximum  xe<;»-i  minimum  urMu  >  0.  | 

We  have  proved  that  the  algorithm  solves  the  matrix  rescaling  problem  correctly  and 
completely.  It  is  finite  if  M  is  a  class  1  or  class  2  matrix.  It  is  possibly  infinite  only  if  M  is  a 
class  3  matrix.  Can  one  give  an  upper  bound  on  the  total  number  of  iterations  needed  in  the 
case  of  finite  termination?  Can  one  prove  that  the  algorithm  is  finite  for  class  3  matrices? 
The  answers  will  be  yes  if  we  have  a  positive  answer  to  the  following  open  question. 

Open  Question.  In  the  case  (ISLI)  has  solutions  (equivalently,  M  can  be  rescaled  positive 
definite),  can  one  give  an  upper  bound  on  one  solution  of  (ISLI)  in  terms  of  M?  Namely, 
can  one  find  a  real  number  h(M)  determined  by  M  such  that  there  exists  a  solution  x  of 
(ISLI)  satisfying  ii  <  h(M)  for  all  i? 
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2.4  Computational  Results 

We  have  coded  the  algorithm  in  FORTRAN.  We  use  the  subroutine  MINOS  (from  the 
Systems  Optimization  Laboratory,  Department  of  Operations  Research,  Stanford  Univer¬ 
sity)  to  solve  LP(k)  and  the  subroutine  F02ABF  (from  Department  of  Computer  Science, 
Stanford  University)  to  calculate  eigenvalues  and  eigenvectors.  The  data  were  randomly 
generated  and  the  program  was  executed  on  a  DEC  20  computer  with  the  following  results, 
see  Table  2.1. 


Problem 

dimension 

Number  of 
iterations 

CPU  time 
(seconds) 

3x3 

5 

3.15 

fTx  5 

14 

7.18 

6x6 

9 

4.69 

8x8 

9 

6.26 

l6  x  16 

8 

r~  6.23  j 

Table  2.1 


2.5  Accelerating  the  Convergence 

Algorithm  2.1  solves  the  semi-infinite  linear  program  (DILP)  by  generating  and  solving  a 
sequence  of  linear  programs  LP(Jfc),  for  k  —  0,  1,  . . . .  If  the  algorithm  does  not  stop  at  a 
certain  iteration  k,  then  a  new  column  D(u*+1)Mut+1  is  generated  and  brought  in.  This 
is  a  cut,  [D(ut+,)Mtil+']Ti  >  1,  on  (SILP).  If  we  want  to  accelerate  the  convergence  of 
the  algorithm,  we  have  to  find  ways  to  generate  more  efficient  cuts. 

Let’s  look  at  the  problem  geometrically.  Suppose  the  algorithm  does  not  stop  at  it¬ 
eration  k.  Let  /*(«)  =  uTD(xk)Mu.  Then,  /*(«’)  >  0,  »  =  1,  ...,  k  and  /*(ufc+1)  <  0. 
Since  /jt(u)  is  a  continuous  function,  for  each  u*, »  =  1,  . . . ,  k,  there  exists  a  relatively  open 
neighborhood  AT*(u‘)  C  S’”-1  such  that  /*(u)  >  0  for  all  u  £  #*(«•).  fk(uk+^)  <  0  means 
that  U«=,  W*(u‘)  does  not  cover  Sn~' .  Suppose  Af  can  be  rescaled  positive  definite  and 
the  algorithm  stops  at  iteration  k.  That  means  U!=i  N *(u* )  covers  Sn~'  ■  Therefore,  we 
want  to  make  #*(«')  bigger  so  that  we  need  fewer  #*(«*)  to  cover  5n_1.  Let’s  consider 
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the  “cut”  [£>(ufc+1)Mtifc+1]rx  >  a  for  some  a  >  1,  and  hope  that  it  will  give  a  bigger 
Nt+\(u').  However,  since  we  are  solving  linear  programs,  changing  all  the  cost  coefficients 
to  a  will  result  in  a  solution  ax*'H  and  therefore  has  no  influence  on  choosing  u*+2.  If  we 
go  over  the  proofs  of  Theorems  2.1  and  2.2,  we  find  that  if  we  change  [D(uk)Muk]Tx  >  1 
to  [D(uk)Muk)Tx  >  at  ,  where  0  <  6  <  at  <  L  for  all  k,  then  Theorems  2.1  and  2.2  still 
hold.  Since 

/jt(u*+1)  =  (l/2)ut+1[D(x*)M  +  MrD(xk))uk+' 

=  (l/2)A[0(x*)Af  +  MrD(x*)], 
if  ft(uk+')  <  0,  a  natural  way  to  choose  a*+i  is 


a*+i  =  -0fk(uk+')  =  -0X  [D(xk)M  +  MrD(xk)], 


where  6  is  a  positive  constant  (if  —  0/*(ut+1)  <  6,  just  let  a *+1  =  6). 

A  number  of  randomly  generated  problems  were  computed  using  the  above  idea  with 
6  —  2  (revised  method)  and  compared  with  a  =  1  (original  method),  see  Table  2.2. 


Problem 

dimension 

No.  of  iterations 
original  method 

5x5 

14 

2 

6x6 

9 

8 

__  gx  8  ~ 

9 

2 

16  x  16 

8 

3 

Table  2.2 


Chapter  3 


Positive  Definite  Least  Square  Estimations 


3.1  Introduction  and  Problem  Formulation 

We  solve  the  following  matrix  estimation  problem: 

Given  two  sequences  of  vectors  a1  and  b 1  in  Rn  with  t  =  1,  . . . ,  L,  a  small  positive  number  e 
and  a  large  number  K  >  >  e,  find  a  real  symmetric  matrix  X  =  (xl} )  such  that  || AT  a'  — 

fc#||2  is  minimized  among  all  the  real  square  matrices  X  satisfying  conditions  (3.1)  and  (3.2): 

XT  =  X  and  -  K  <  xt}  <  K  for  all  i,j  =  1, . . . ,  n.  (3.1) 

The  smallest  eigenvalue  of  X  is  no  less  than  e.  (3.2) 

This  problem  arises  from  mathematical  economics  (see  Dantzig,  McAllister  and  Stone 
(1988)  and  Lau  (1978)).  It  differs  from  the  ordinary  least  square  estimation  in  that  the 
estimated  matrix  X  is  required  to  be  positive  definite  (Fact  2.7).  One  way  of  solving  this 
kind  of  problem  was  given  by  Lau  (1978).  He  transformed  the  problem  into  an  unconstrained 
nonlinear  least  square  problem  by  matrix  factorization  and  suggested  that  the  latter  can 
be  solved  by  any  algorithm  designed  for  unconstrained  nonl'ueai  programs.  However,  since 
the  objective  function  for  his  transformed  problem  is  not  convex,  global  convergence  is 
not  guaranteed.  Herein  we  transform  the  problem  into  an  equivalent  semi-infinite  convex 
quadratic  program  and  develop  a  method  to  solve  it.  The  solution  method  essentially  solve 
the  problem  by  ignoring  the  positive  definitness  condition,  then  check  to  see  for  the  solution 
X  =  X°  whether  there  is  a  vector  u  =  u°  such  that  u°rA°un  <  e.  If  there  is,  the  linear 
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inequality  in  X,  namely  u°  Xu°  >  a  >  e  become  an  additional  constraint  that  X  must 
satisfy  and  the  problem  is  iteratively  solved  again  obtaining  X  =  Xk  until  on  some  major 
iteration  k  no  vector  u  =  uk  can  be  found  such  that  uk  Xkuk  <  e. 

As  in  Chapter  2,  S ”_1  =  {a:  €  Rn  ■  xrx  =  1}  denotes  the  unit  sphere  in  Rn  and  ||x|| 
denotes  the  Euclidean  norm  of  x.  For  a  real  symmetric  matrix  B ,  A[Z?]  stands  for  the  small¬ 
est  eigenvalue  of  B  and  V[B\  a  corresponding  eigenvector  of  unit  length.  Superscripts  on 
vectors  are  used  to  denote  different  vectors,  while  subscripts  are  used  to  denote  components 
of  a  vector. 

To  solve  this  estimation  problem,  we  first  transform  it  into  an  equivalent  (vector  form) 
semi-infinite  convex  quadratic  program. 

For  the  given  data  a* ,  b1  £  Rn  for  t  =  1,  . .  . ,  L,  let  A  =  (a1  •  •  ■  ar  )T  and  B'  = 
(bj  ■  ■  ■  b[  )T  for  all  i  =  1,  ... ,  n.  Then  A  is  an  L  x  n  matrix,  Bl  is  an  L- vector,  and  ArA  is 
an  n  x  n  square  matrix.  Let  M  be  an  n2  x  n2  block-diagonal  matrix  with  diagonal  blocks 
Ar A  and  E  be  an  nL  x  n2  block-diagonal  matrix  with  diagonal  blocks  A,  namely, 

(ATA  0  •••  0  \ 

I  0  AT  A  ■■■  0  1 


A  0 
0  A 


Let  Xt.  be  the  >-th  row  of  matrix  A'  for  all  i  =  1,  . . . ,  n,  and  Y  =  F(X)  =  (Aj.,  •  ■  •  ,  Xn.)T  = 
(*ii,  •  •  •  ,*in,  ■  •  For  example,  if 


\  2  3 
X  =  4  5  6 

\7  8  9 


then  Y  =  F(X)  =  (1, 2, 3,4,5, 6, 7, 8, 9)r.  In  terms  of  vectors,  condition  (31)  becomes: 


i—  i )n+j  —  x,j  —  x j ,  —  )n+j  for  all  i , )  —  1 , .  ■  • ,  n 
and  —  1\  <  Yu  <  K  for  all  i  =  1, .  . . ,  n2. 


(3.1') 
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By  Fact  2.6,  condition  (3.2)  is  equivalent  to:  uTXu  >  e  for  all  u  £  Sn  1 .  Therefore, 
condition  (3.2)  becomes: 

n 

uTXu  =  uiUT(Xi.)T  =  (u}uT  . . .  u„ut)Y  >  c  for  all  u  £  5n_1  -  (3-2') 

>=i 

The  objective  function  becomes: 

2l|Aro'-6'||2  =  £E(.ViV-6')J 

<=1  (=1  »=1 

=  E 

1=1  1=1 

=  EX>'r**r-6«>2 

1=1  1=1 

=  2M(Xi.)t-B*||2 

i=i 

=  ArA(Xt.)T  -  2 (B')TA(Xt.)T  +  (B')rB'} 

1=1 

n 

=  Yr MY  ~  2({B')T  ■  ■  ■  {Bn)r)EY  +  '$2{B')T Bl . 

t=i 

Consequently,  the  equivalent  semi-infinite  quadratic  program  is 

(SIQP)  minimize  YrMY  -  2((fl1)r  -{Bn)T)EY  +  E^B*)7’#* 
subject  to 

(u,  uT  •  •  ■  unur)Y  >  c  for  all  u  €  S""1 


^(t  —  1)n+j  ^(j  —  l)n.+  i  —  0  1,  .  .  .  ,  M 

-K  <  Y,  <  K  for  all  i  =  1, . . . ,  n2. 


Remark  3.1. 

(1)  M  is  positive  semidefinite  because  each  of  its  diagonal  blocks  ArA  is  positive 
semidefinite.  This  implies  that  the  objective  function  of  (SIQP)  is  convex.  The  feasible 
region  of  (SIQP)  is  compact  and  convex,  and  is  defined  by  an  infinite  number  of  linear 
constraints.  Moreover,  it  is  nonempty  since  Y  =  F(el)  is  a  feasible  solution,  where  I  is  the 
identity  matrix.  Therefore,  (SIQP)  is  a  feasible  semi-infinite  convex  quadratic  program  and 
optimal  solutions  for  (SIQP)  exist. 
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(2)  Since  Y  =  F(X)  =  (X^.  ■  Xn.)  is  a  bijection,  F~'(  • )  exists.  Therefore,  X  — 

F_1(V)  and  uTXu  =  urF~'(Y)u  =  (ttiur  •  •  •  unuT)Y.  For  convenience,  we  use  uT F~'(Y)u 
and  (u,ur  •  •  •  unuT)Y  interchangeably. 

We  have  shown  that  this  positive  definite  least  square  estimation  problem  is  equivalent 
to  the  semi-infinite  convex  quadratic  program  (SIQP).  In  Section  3.2,  we  propose  an  algo¬ 
rithm  for  solving  (SIQP)  and  prove  its  convergence.  In  Section  3.3,  we  estimate  the  total 
number  of  major  iterations  and  the  final  objective  function  value  to  attain  a  prescribed  level 
of  approximation  to  an  optimal  solution.  In  Section  3.4,  we  present  computational  results 
for  randomly  generated  data. 

3.2  An  Algorithm  and  its  Convergence 

We  propose  an  algorithm  for  solving  the  semi-infinite  quadratic  program  (SIQP).  This 
algorithm  solves  (SIQP)  by  generating  and  solving  a  sequence  of  feasible  convex  quadratic 
programs  QP(F)  for  k  =  1,  2,  . . . .  Every  QP(£)  has  the  same  objective  function  as  that  of 
(SIQP)  and  the  feasible  region  of  QP(fc)  contains  that  of  QP(fc  +  1). 

Algorithm  3.1. 

Step  1  (Initialization). 

Let  k:  =  0; 

let  a  be  a  constant  such  that  e  <  a  <<  A'; 
let  QP(0)  be  the  quadratic  program 

minimize  H(Y)  =  YTMY  -  2((B')r  ■  (Bn)r)EY  + 

subject  to 

Y( i— i )n+j  (j — —  0  for  &11  i i j  —  l,...,n 

-K  <  Y,  <  K  for  all  i=  1 . n2. 

Step  2. 

Find  an  optimal  solution  Yk  of  QP(fc); 

let  X*  =  F~'(Yk ),  i.e„  xki}  =  Yk_V)n+j  for  all  ij  =  1 . n; 
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calculate  A[X*]  and  VjX4]; 
if  A[A*]  >  e,  go  to  Step  4. 

Step  3. 

Let  uk  =  V[Xfc]; 

form  QP(fc  +  1)  by  adding  a  cut,  ( uk)TF~'(Y)uk  >  a  ,  to  QP(k); 
k  :=  i  +  1; 
go  to  Step  2. 

Step  4. 

If  a  >  e,  Yk  is  an  approximate  optimal  solution  of  (SIQP);  stop. 

If  a  =  e,  Yk  is  an  optimal  solution  of  (SIQP);  stop. 

Comments  on  Algorithm  3.1. 

(1)  For  any  k,  the  feasible  region  of  QP(ifc)  is  a  nonempty  polytope  since  Y  =  F(al)  is 
a  feasible  solution.  Therefore,  optimal  solutions  exist  for  all  QP (k).  Furthermore,  since  the 
objective  function  of  QP(Ar)  is  quadratic  and  convex,  there  are  finite  algorithms  for  finding 
one  of  its  optimal  solutions. 

(2)  Efficient  finite  algorithms  for  calculating  eigenvalues  and  eigenvectors  of  a  matrix 
to  any  specified  level  of  approximation  can  be  found  in  Wilkinson  (1965)  - 

(3)  k  counts  the  number  of  major  iterations  (Step  2-Step  3),  or  equivalently,  the  number 
of  cuts  added  before  termination.  Each  major  iteration  can  be  processed  finitely. 

(4)  a  is  a  constant  and  e  <  a  <<  K.  If  a  >  e,  then  an  approximate  optimal  solution 
will  be  found  after  a  finite  number  of  major  iterations  (see  Theorem  3.1).  If  a  =  e,  then 
any  cluster  point  of  of  the  sequence  V0,  V1 ,  V2,  ...  is  an  optimal  solution  for  (SIQP)  (see 
Theorems  3.2  and  3.3). 

Theorem  3.1.  If  a  >  e,  then  Algorithm  3.1  can  find  an  approximate  optimal  solution  of 
(SIQP)  after  a  finite  number  of  major  iterations. 

Proof.  Let  H(Y)  =  YT MY  -  2 ((B')T  ■  ■ -{Bn)T)EY  +  £?=! {B')T{B')  be  the  objective 
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functions  of  (SIQP)  and  QP(ifc)  for  all  k.  Let 

C  =  {Y  :  -K  <  Yi  <  K,i=  1 . n2}  x  Sn~'  and  G(Y,u)  =  uTF~'(Y)u. 


Then  G(Y,  u)  is  uniformly  continuous  on  C ■  Therefore,  for  6  =  a  —  e  >  0,  there  exists  17  >  0 
such  that 


||(Y,u)  -  (Y,u)||  <  T)  implies  |G(Y,tz)  -  G(Y,u)|  <  6 


(3-3) 


for  all  (Y,  u)  and  (Y,  u)  in  C. 

If  Algorithm  3.1  goes  on  infinitely,  then  it  generates  a  sequence  u*  £  Sn~ 1  for  k  =  0,  1,  2, 

- Since  S,"-,  is  compact,  for  the  tj  >  0  ,  there  exist  uk‘  and  in  the  sequence  such  that 

||  <  t).  Without  loss  of  generality,  we  assume  that  Jbj  <  Jfcj  .  Since  Algorithm  3.1 
does  not  stop  at  iteration  kj  and  ki  <  ifcj,  we  have: 


G(Yki ,  uk<)  =  (t tk‘)TF-\Yki)uki  >  a;  (3.4) 

G{Yk\uk>)  =  (uk>)TF-'(Yki)uk*  <  e.  (3.5) 

However,  (3.4)  and  (3.5)  contradict  (3.3).  Therefore,  if  a  >  e,  then  Algorithm  3.1  must 
terminate  finitely.  Suppose  that  it  stops  at  a  certain  iteration  k.  Then,  A[F_1(Yfc)]  >  e 
and  by  Fact  2.6,  Yk  is  a  feasible  solution  of  (SIQP).  However,  since  a  >  e,  the  constraints 
(ufr'fyy  >  a  for  *  =  0,  1,  . . .  ,  k  —  1  may  be  violated  by  certain  feasible  solutions 
of  (SIQP).  We  can  not  guarantee  that  H(Yk)  <  H(Y)  holds  for  all  feasible  Y.  But,  if  a 
feasible  solution  of  (SIQP)  satisfying  A(F_1(Y)]  >  a,  then  H(Yk)  <  H(Y)  is  guaranteed 
to  hold.  Therefore,  Y*  is  only  an  approximate  optimal  (or  a-suboptimal)  solution  in  the 
case  a  >  e.  | 


Remark  3.2.  When  a  increases,  the  number  of  cuts  required  to  be  added  before  termina¬ 
tion  decreases,  but  the  final  objective  function  value  increases.  For  given  data  and  choice 
of  a,  if  the  algorithm  does  not  terminate  by  a  specified  number  of  iterations,  we  can  always 
increase  a  and  try  again. 

Theorem  3.2.  If  a  =  e  and  Algorithm  3.1  stops  at  a  certain  iteration  k,  then  Yk  is  an 
optimal  solution  of  (SIQP). 
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Proof.  Let  H(Y)  be  the  objective  functions  of  (SIQP)  and  QP(t)  for  all  i.  If  Algorithm  3.1 
stops  at  a  certain  iteration  k,  then  A[X*j  >  e.  By  Fact  2.6,  V*  is  a  feasible  solution  of 
(SIQP).  Next,  suppose  that  Y  is  an  arbitrary  feasible  solution  of  (SIQP).  Because  a  =  e, 
all  cuts  generated  in  Step  3  are  necessary.  Hence,  Y  is  feasible  for  all  QP(i).  1°  particular, 

Y  is  feasible  for  QP(ib).  Since  Yk  is  an  optimal  solution  of  QP (fc),  we  have  H(Yk)  <  H(Y). 
It  follows  that  Yk  is  an  optimal  solution  of  (SIQP).  | 

Theorem  3.3.  Suppose  that  a  =  e  and  that  Algorithm  31  does  not  stop  finitely.  Let  the 
sequence  Y°,  V1,  Y2,  ...  be  generated  by  Algorithm  3.1.  Then  any  cluster  point  of  this 
sequence  is  an  optimal  solution  of  (SIQP). 

Proof.  Let  Yk  and  uk  for  k  =  0,  1,  2,  . . .  be  generated  by  Algorithm  3.1  with  a  =  e.  Since 
Yk  for  k  =  0,  1,  ...  are  in  a  compact  set,  there  exist  cluster  points.  Let  Y*  be  a  cluster 
point,  and  without  loss  of  generality  we  assume  that  lim*-.oo  Yfc  =  Y*.  We  claim  that  for 
any  /?  satisfying  0  <  /?  <  e,  there  exists  an  integer  N{&)  such  that  A[F-1  (Y*)]  >  /?  for  all 
k  >  N((f).  We  skip  the  proof  here  since  it  is  similar  to  the  proof  in  Theorem  3.1.  By  this 
claim  and  the  continuity  of  F-1(-)  and  A[  ]  (Fact  2.7),  we  have  A[F_1(Y*)]  >  /?  for  any 
/3  satisfying  0  <  /?  <  c.  Consequently,  A[F_1(Y*)]  >  e  and  Y*  is  feasible  for  (SIQP).  It 
remains  to  show  that  Y*  is  optimal.  Let  Y  be  an  arbitrary  feasible  solution  of  (SIQP).  Then 

Y  is  feasible  for  all  QP (k).  Because  Yk  is  an  optimal  solution  of  QP(fc)  and  the  objective 

functions  H(Y)  of  QP (k)  and  (SIQP)  are  the  same,  we  have  H(Yk)  <  H(Y)  for  k  =  0,  1, 
2, _ It  follows  that  H(Y*)  <  H(Y)  holds  for  all  feasible  Y  and  Y*  solves  (SIQP).  | 

Remark  3.3.  Since  a  =  e,  the  feasible  region  of  QP (k)  contains  that  of  (SIQP).  As  the 
algorithm  goes  on,  Y*  becomes  more  and  more  close  to  the  feasible  region  of  (SIQP)  and 
H{Yk)  tends  increasingly  to  H(Y*). 

3.3  Estimation  of  Upper  Bounds 

We  have  proved  that  if  a  >  e,  then  the  algorithm  can  find  an  approximate  optimal  solution 
of  (SIQP)  after  finitely  many  iterations.  Let  Y*  denote  an  optimal  solution  of  (SIQP)  and 
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Yn  denote  the  approximate  optimal  solution  for  some  a  >  e.  We  know  that  Yn  is  feasible 
for  (SIQP)  and  that  for  any  feasible  Y  satisfying  A[F-’(y)]  >  a,  H{Y)  >  H{Yn)  >  H(Y *) 
holds.  Now  we  want  to  know  how  good  Yn  is,  namely,  we  want  to  have  an  upper  bound 
for  \H(Yn)  —  H(Y*)\.  Remember  that  when  a  =  e,  the  algorithm  generates  a  sequence 
V*  for  i  s  1,  2,  and  H(Y *)  tends  increasingly  to  H(Y*).  Therefore,  if  we  want  to 
know  how  good  y**  is,  we  reset  a  =  c  and  execute  the  algorithm  until  some  iteration  fc. 
Then  we  have  | H(Yn)  -  i/(y*)l  <  H{Yn)  -  H(Yk),  and  |tf(y")  -  H(Y*)\/H(Y*)  < 
( H(Yn )  -  H(Yk))/H(Yk). 

In  the  rest  of  this  section,  we  give  an  upper  bound  of  the  total  number  of  major 
iterations  needed  for  finding  Yn .  We  begin  with  Lemma  3.1,  which  was  stated  and  proved 
by  Alan  J.  Hoffman  (verbal  communication). 


Lemma  3.1.  If  u,  u  £  Sn  1 , 


then  E.n,j=t  l(«»  -  «»KI  <  nllu  -  «ll- 


Proof.  Let  u;  =  (u;  -  u;|  for  t  =  1,  ...,  n,  r1  =  (n, x 2  =  (t/2,  . . . ,  t/n,  tq  ),  .... 
Xn  =  (wn,Wi.  ■  •  -  ,Wn-i),  and  y  =  (|u,|,  ...,|u„|).  Then, 

n  n 

|(u<  -  u,)uj|  =  Y  v*lu>l 

«,j=i  «,j=i 

=  yrx'  +  yTx2  + - f  yTxn 

<Hy|lElMI 

i=i 

=  "||y||||*1|| 

=  n||«  -  “II-  I 


Lemma  3.2.  Suppose  that  |ju  —  uj|  <  (a  -  f)/2 nK  where  u,  u  €  Sn~x ,  a  >  e  >  0,  and  K 
is  the  given  bound  on  Y.  IfY  is  feasible  for  (SIQP)  and  G(Y,u)  >  a,  then  G(Y,u)  >  e. 
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Proof. 

n 

I G(Y,u)  -  <?(K,u)|  =  |  (u»uj  ~ 

n 

<  #  i(ui  -  -  «>)i 

<  2n/'i||u  -  ujj  (by  Lemma  3.1) 

<  a  —  e. 

Therefore,  G(Y,  u)  >  G(Y,  u)  -  (a  -  c)  >  a  -  (a  -  e)  =  e.  | 

Let  Bn( u,  r)  denote  the  n~dimensional  Euclidean  ball  with  center  u  G  Rn  and  radius 
r.  At  iteration  k,  G(Y,  uk)  >  a  is  added  into  the  constraints  of  QP(fc  +  1).  If  the  algorithm 
does  not  stop,  then  it  will  find  a  ufc+1  G  5n-1  satisfying  G(Tfc+1,u*+1)  <  e.  Since  y*+1 
is  feasible  for  QP(fc  4-  1),  we  have  G(Vk+1 ,  u')  >  a  for  t  =  1,  . . . ,  Jk.  From  Lemma  3.2,  we 
know  that  for  all  k  =  1,  2,  . . . , 

k 

“*+1  t  (J  Bn(u\  (a  -  e)/2nK)  n  Sn~' .  (3.6) 

i=1 

Hence,  if  at  a  certain  iteration  k  we  have 

k 

U  fln(u',  (a  -  e)/2 nl<)  n  Sn~'  =  Sn~ 1 ,  (3.7) 

i=) 

then  the  algorithm  must  stop  since  it  cannot  find  a  u*+1  G  5”-1  satisfying  G(y*+1 ,  ufc+1)  < 
e.  To  estimate  a  k  such  that  (3.7)  holds,  we  need  to  know  a  lower  bound  of  the  (n  —  1)- 
content  (volumn)  of  UjLjBnfti*,  (a  —  e)/2nK)  DS’’1-1 .  In  order  to  obtain  this  lower  bound, 
we  need  a  lower  bound  of  the  (n  -  l)-content  of  Bn(u ,  r)n5"-1 .  In  the  following  discussion, 
we  use  Vj(T)  to  denote  the  j-content  of  a  j-dimensional  set  T. 

Fact  3.1. 

(1)  Vn(Bn{u,r))  =  rn>/7r”/r((n/2)  +  1), 

(2)  yn_,(5n-,)  =  2v^/r(n/2), 

where  T(/?)  =  x^~ye~Tdx  for  j3  >  0  is  the  gamma  function. 
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Lemma  3.3.  If  u  €  Sn  1  and  r  <  y/2  ,  then, 

Vn-i(Bn(u,r)DSn-')  >  (V^72r)»-Vr((n  +  l)/2). 

Proof.  Without  loss  of  generality,  let  u  =  (0, 0, . . . ,  I).  Then, 

(B„(u,r))=  /••  /  (l“*i - -  din-, 

>  I  “  I  dx\  •  •  •  dxn-i , 

_ ^3  ,</>* 

where  p  is  obtained  by  solving  the  following  system  of  equations: 

A  +  •  •  •  +  A  =  1 

*i+--+*n-i  +(l-*n)2  =  p2  (3.8) 

*?  +  •••  +  *’_,  =P2. 

Solving  (3.8)  we  get  p 2  =  r2(4  —  r2)/ 4.  Since  r  <  v^2,  we  have  />  >  r/>/ 2.  Therefore, 

Vn_1(S,l(u,r)nS’l-1)>  J-J  dzi--dxn-, 

*?+—+*»_,  <r/v^ 

=  (v/V2r)"-Vr((n  +  l)/2).  | 

Theorem  3.4.  If  a  >  e,  then  an  upper  bound  for  the  total  number  of  major  iterations 
needed  for  finding  Y"  is:  (ir/2)y/n  (4v/2nA'/(a  -  e))n_1. 

Proof.  From  (3-6)  we  know  that  Bn(u',  (a  -  e)/4nK)  D  Bn(uJ,(a  —  t)/4nK)  is  empty  for 
all  *  ^  j.  Therefore, 

k 

Vn-,  ((J  fln(u‘,  (a  -  t)/2ntf)  n  S""1) 

i=i 

k 

>  K„_,  (U  Bn(u*,  (a  -  e)/4nK)  n  S’*"1 ) 

i=i 

=  (£„(«’,  (a  -  f)/4n/\ )  n  Sn_1) 

>  fc(V^72(or  -  e)/4n/v))”-1/r((n  +  l)/2). 


(3.9) 


Section  3.4 


Computational  Results 


30 


It  follows  from  (3.7),  (3.9),  and  Fact  31  that  an  upper  bound  for  the  total  number  of  major 
iterations  for  finding  Ya  is: 

_ 2y^F/r(n/2) _ =  2vAF(4^n/Q"~1r((n  +  l)/2) 

(y/7/2(a-e)/4nK)n~'/r((n  +  l)/2)  (a  -  e)""1  T(n/2) 

Since  G(Y,  u)  =  G(— Y,  u)  for  all  u  G  ft",  only  half  of  Sn-1  needs  to  be  covered  by 
U*=i  Bn(u* ,  (a  —  e)/2 nK  D5n_1).  Therefore,  we  can  cut  the  bound  in  (3.10)  by  half. 
Also,  it  is  not  hard  to  show  that  T((n  -f  l)/2)/r(n/2)  <  \/n¥/2  (see  Appendix  A).  Thus 
Theorem  3.4  is  established.  | 

3.4  Computational  Results 

We  have  coded  Algorithm  3.1  in  FORTRAN.  We  use  the  subroutine  QPSOL  (from  the  Sys¬ 
tems  Optimization  Laboratory,  Department  of  Operations  Research,  Stanford  University) 
to  solve  QP(Jfc)  and  the  subroutine  F02ABF  (from  Department  of  Computer  Science,  Stan¬ 
ford  University)  to  calculate  eigenvalues  and  eigenvectors.  The  program  was  executed  on  a 
DEC  20  computer.  All  components  of  the  input  data  a 1  and  b1  are  i.i.d.  U{— 0  5, 0.5).  First, 
we  compute  one  problem  six  times  with  different  a  values  to  demonstrate  the  influence  of 
a  on  the  number  of  major  iterations  and  on  the  final  objective  values,  see  Table  3.1. 

Given  data: 

L  =  8,  6  =  1.0,  A  =  10",  n2  =  16; 

a1  =  (-0.3052,  0.1087,  -0.3915,  -0.4383) 

a2  =  (  0.1379,  0.1707,  -0.1208,  0.3839) 

a'1  =  (  0.2999,  -0.4803,  0.1790,  -0.2021) 

a4  =(-0.1334,  0.1864,-0.0431,  0.4557) 

aR=  (-0.0681,  -0.4827,  -0.1384,  0.0547) 

a6  =(-0.4691,  0.0743,  0.3823,  0.1650) 

a7  =(-0.2117,  -0.3549,  0.4991,  -0.1264) 

aR=  (-0.0885,  0.0886,  -0.4886,  -0.3304) 

b1  =  (  0.2325,  -0.1774,  -0.3115,  0.2133) 
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b 2  =  (-0.4512, 
fc3  =  (-0.0641, 
b4  =  (-0.3645, 
t5  =  (-0.2327, 
66  =  (-0.3909, 
b7  =  (-0.1478, 
6*  =  (-0.2671, 


-0.1078,  0.0383,  -0.0906) 

-0.3664,  -0.1086,  -0.3182) 
-0.1941,  -0.1331,  -0.3830) 
-0.0301,  -0.0613,  0.2470) 

0.3732,  -0.0953,  -0.1953) 
-0.2652,  -0.3996,  0.3307) 

0.3283,  0.0569,  -0.3668) 


Value  of 
alpha 

No.  of  major 
iterations 

Final  objective 
function  value 

CPU  time 
(seconds) 

1.1 

4 

537T3 

1.89 

1.01 

7 

4.8017 

2.77 

1.001 

13 

4.7585 

4.94 

1.0001 

14 

4.7537 

5.44 

1.00001 

15 

“  4.7531 

5M 

)  1. 00060 r 

16 

4.7532 

h  6.63  1 

Table  3.1 


Next  we  solve  a  number  of  problems  in  different  dimensions,  see  Table  3  2.  Again,  all 
components  of  the  input  data  a*  and  6 *  are  i.i.d.  (7(— 0.5, 0  5).  a  =  1.01,  e  =  1,  and  problem 
dimension  =  [n2,L], 


Problem 

dimension 

No.  of  major 
iterations 

Final  objective 
function  value 

CPU  time 
(seconds) 

[16,6] 

7 

3.3775 

2.9 

■OESiOlB 

8 

5.9837 

3.32 

mmtwwm 

15 

9.0326 

25.18 

rz 

36,12; 

17 

14.1864 

28.19 

i  PS 

33 

19.1114 

1  [64,181 

26 

29.6169 

175.32 

Table  3.2 


A  practical  problem  based  on  economic  data  was  al«o  solved.  It  was  quickly  solved  in 


two  major  iterations. 
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4.1  Introduction  and  Preliminaries 

The  primal  problem  (SIL)  of  semi-infinite  linear  programming  is  defined  as 

(SIL)  minimize  crx 
subject  to 

a(u)ra:  —  b(u)  >  0  for  all  u  €  U , 

where  c,  x  £  Rn,  U  C  Rm  is  an  infinite  parameter  set,  a  :  U  — >  Rn,  and  b  :  U  — *  ft1.  The 
dual  of  (SIL)  is:  for  a//  finite  subsets  {u*  :  i  €  A}  of  U 

(SID)  maximize  &(«')!/* 

subject  to 

£,€a  a(u')y>  =  c 

A  is  finite  and  {u‘  :  :  £  A)  C  (/ 
yi  >  0  for  all  i  £  A  . 

We  can  always  rewrite  (SID)  in  the  format  of  generalized  linear  programming  (see 
Dantzig  (1963)  Chapter  22)  as  follows: 
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(GLP)  maximize  yo 
subject  to 

Poyo  +  py  =  q 

p  may  be  freely  chosen  from  S 

y  >  0, 

where  S  =  con v{(a(u)T ,  —b(u))T  :  u  G  U },  po  =  (0, . . .  ,0, 1),  and  q  ~  (cr,0)r.  Indeed,  for 

any  {u*  :  i  G  A}  C  U  with  finite  A  and  any  {y,  >  0  :  i  G  A}  satisfying  a(ti*)yi  =  c, 

let  y0  =  ICigA  Ku*)jh.  y  =  E.6A  y*  >  o  (since  c  ^  0)  and  p  =  ^2,eA(a(it')T,  -6(u*))ry,/y. 

* 

Then,  p  £  S  and  p0yn  +  py  =  q.  Conversely,  if  pnyn  +  py  =  q,  where  y  >  0  and  p  G  5, 
then  there  exist  ti‘  G  U  and  A,  >  0  for  i  =  1,  . . . ,  t  (t  <  n  +  1)  such  that  *»  =  1 

andp  =  X2Li(a(u*)T> — (see,  e.g.,  Rockafellar  (1972)  p.155).  Let  y,  =  A,y  >  0  for 
»  =  1,  ■  ■ t.  Then,  £  ■_,  a(ti‘)y<  =  c  and  yn  =  b(u’)yt.  Hence,  (SID)  and  (GLP)  are 
equivalent. 

The  first  algorithm  for  solving  (GLP)  and  therefore  (SID)  was  given  by  Dantzig  and 
Wolfe,  see  Dantzig  (1963),  Chapters  22  and  24,  and  Dantzig  (I960).  Starting  with  a  non¬ 
degenerate  basic  feasible  solution  of  (GLP),  the  Dantzig- Wclfe  algorithm  solves  (GLP)  by 
generating  a  pk  with  the  smallest  negative  reduced  cost  at  each  iteration  ib  and  then  solves 
the  new  restricted  master  program.  Although  one  can  start  this  algorithm  with  any  basic 
feasible  solution,  the  nondegeneracy  of  the  starting  basic  feasible  solution  is  required  by  the 
convergence  proof.  How  to  find  a  starting  nondegenerate  basic  feasible  solution  is  still  an 
open  question. 

Gustafson  and  Kortanek  proposed  an  algorithm,  the  alternating  procedure ,  for  solving 
the  primal  semi-infinite  linear  program  (SIL)  (see  Gustafson  and  Kortanek  (1973))  under 
the  following  assumptions: 

(Al)  U  is  compact, 

(A2)  a(u)  and  b(u)  are  continuous, 

(A3)  the  feasible  region  of  (SIL)  is  nonempty, 

(A4)  the  feasible  region  of  (SIL)  is  contained  in  T,  where  T  C  Rn  is  a  nonempty 
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poly  tope. 

The  alternating  procedure  solves  (SIL)  as  follows:  let  x°  be  an  optimal  solution  of  the  linear 
program 

minimize  crx 
subject  to 

x€T. 

At  iteration  k,  if  a( u)T xk  —  b(u)  >  0  for  all  u  G  U ,  then  xk  is  an  optimal  solution  of  (SIL) 
and  the  procedure  is  stopped.  Otherwise,  find 

u*+1  €  argmin{a(u)Ta:  —  b(u)  :  u  G  U} 

and  let  r*+1  be  an  optimal  solution  of  the  linear  program 
minimize  crx 
subject  to 

a(u‘)Tx  —  6(u‘)  >  0  for  all  i  =  1,  . . . ,  k  +  1 
*<=T. 

Under  the  above  assumptions,  Gustafson  and  Kortanek  (1973)  proved  that  cTx*  =  v(SIL), 
where  x*  is  a  cluster  point  of  the  sequence  xk  generated  by  the  alternating  procedure  and 
v(SIL)  stands  for  the  optimal  objective  function  value  of  program  (SIL).  The  convergence 
proof  of  the  alternating  procedure  provided  by  them,  however,  is  incomplete:  they  did  not 
prove  the  Feasibility  of  x *  (although  it  can  be  shown  that  x*  is  a  feasible  solution  by  the 
same  approach  as  the  proof  of  Theorem  4.1). 

Herein  we  present  a  one-phase  algorithm  for  solving  a  large  class  of  semi-infinite  lin¬ 
ear  programs.  This  algorithm  is  based  on  Algorithm  2.1  and  Algorithm  31  and  can  be 
generalized  to  solve  nonlinear  semi-infinite  programs.  It  has  several  advantages: 

1.  It  handies  feasibility  and  optimality  together  and  thus  eliminates  the  extra  work  of 
finding  a  starting  feasible  solution  that  might  be  far  away  from  an  optimal  solution. 

2.  It  can  detect  infeasibility  after  a  finite  number  of  iterations. 

3.  It  does  not  impose  any  restriction  on  (/,  o(u ),  and  £>(u),  i.e.,  U  could  be  any  nonempty 
set  in  Rm ,  a(u)  could  be  any  vector  in  /?”,  and  6(u)  could  be  any  real  number. 
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4.  The  cut  it  generates  at  each  iteration  need  not  be  the  most  violated  cut  or  near  the 
most  violated  cut.  The  strength  of  the  cuts  can  be  controlled  by  a  parameter. 

5.  It  solves  the  primal  and  the  dual  programs  simultaneously. 

The  rest  of  this  chapter  is  organized  as  follows.  In  Section  4.2  we  present  an  algorithm 
for  solving  a  large  class  of  semi-infinite  programs  and  prove  its  convergence  in  two  steps. 
First,  we  show  that  it  can  find  an  e-optimal  solution  after  finitely  many  iterations,  and  then 
use  this  result  to  show  that  it  can  find  an  optimal  solution  in  the  limit.  In  Section  4.3 
we  prove  a  duality  theorem  and  show  that  the  algorithm  solves  the  primal  and  the  dual 
programs  simultaneously.  In  Section  4.4  we  estimate  how  good  an  e-optimal  solution  is 
compared  to  an  optimal  solution  and  give  an  upper  bound  on  the  total  number  of  iterations 
needed  for  finding  an  e-optimal  solution  under  certain  assumptions.  In  Section  4.5  we  show 
that  the  algorithm  can  be  generalized  to  solve  a  class  of  nonlinear  semi-infinite  programming 
problems. 

4.2  An  Algorithm  for  Semi- Infinite  Linear  Programming 

We  first  present  a  one-phase  algorithm  for  solving  the  primal  problem  (SIL)  of  semi-infinite 
linear  programming  that  satisfies  the  assumption  that  the  feasible  region  is  contained  in  a 
polytope  T. 

Algorithm  4.1. 

Step  1  (Initialization). 

Let  k  :=  0; 

let  a  be  a  constant  such  that  0  <  a  <  1; 
let  LP(0)  be  the  linear  program 
minimize  cTx 
subject  to 

x  €  T. 

Step  2. 

If  LP(fc)  is  infeasible,  then  (SIL)  is  infeasible,  stop. 
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Else,  find  an  optimal  solution  xk  of  LP(&); 

if  a(u)rxk  —  b{u)  >  0  for  all  u  £  U , 

then  xk  is  an  optimal  solution  of  (S1L),  stop. 

Step  3. 

Find  a  uk  £  U  such  that 

either  a(uk)Txk  —  b(uk)  <  —  a 

or  a(uk)T  xk  —  b(uk)  <  a  ■  inf{a(u)rxfc  —  b(u)  :  u  €  U}  ; 
form  LP(ifc  +  1)  by  adding  a  cut,  a(uk)n  x  —  b(uk)  >  0,  to  LP(fc); 
k:=k+  1; 
go  to  Step  2. 

Comments  on  Algorithm  4.1. 

(1)  We  assume  that,  given  xk ,  there  are  algorithms  that  can  find  an  approximate  solu¬ 
tion  of  the  nonlinear  program  mf{a(ti)rx*  —  b(u)  :  u  £  U).  Namely,  given  any  small  positive 
number  6,  these  algorithms  can  find  a  u  6  U  such  that  a(u)Txk  —  6(u)  <  in f{a(u)rxk  —  b(u)  : 
u  e  U}  +  6. 

(2)  In  Step  2,  to  test  whether  a(u)Txk  -  6(u)  >  0  for  all  u  G  U ,  one  can  proceed  as 
follows:  given  a  small  positive  number  6,  find  a  u  £  U  such  that 

a(u)Txk  —  b(u)  <  inf{a(u)rarfc  —  6(u)  :  u  £  U]  +  6. 

If  a(u)Txk  —  b(u)  >  6,  then  inf{a(u)rx*  —  b(u)  :  u  G  U)  >  0  and  xk  is  optimal.  If 
a(u)Txk  —  6(u)  <  0,  then  inf{a(u)rxt  —  b(u)  :  u  €  U)  <  0  and  one  can  go  to  Step  3. 
Otherwise,  0  <  a( u)Txk  —  6(u)  <  <5,  which  implies  that  inf{a(u)Txk  —  b(u)  :  u  €  U)  >  —S, 
and  therefore,  r*  is  an  ^-optimal  solution  (see  Definition  3.1  below).  If  one  wants  an  exact 
optimal  solution,  then  one  needs  an  algorithm  that  can  find  inf {a(u)rx*  —  b(u)  :  u  £  U} 
starting  from  a  6-optimal  solution, 

(3)  In  Step  3,  either  there  is  a  uk  that  satisfies  a(uk)Txk  —  b(uk)  <  —a  or  else  0  > 
inf{a(u)Tx*  —  6(u)  :  u  £  U)  >  -or  >  —  oe.  In  the  latter  case, 

a  ■  inf{a(u)Txfc  -  6(u)  :  u  €  U }  >  inf{a(u)  rxk  —  b(u)  :  u  £  U}(  since  a  <  1), 
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and  hence  there  exists  a  uk  £  U  such  that 

a(uk)Txk  -  b(uk)  <  a  ■  inf {a(u)Txk  —  b(u)  :  u  £  [/}■  (4.1) 

To  find  a  uk  satisfying  (4.1)  by  any  algorithm  that  has  the  property  stated  in  (1),  a  stopping 
rule  is 

a(uk)Txk  -  b(uk)  <  inf{a(ti)rx*  —  6(u)  :  u  £  U}  4-  6  and 

6<(a-  1  )a(uk)Txk  -  b{uk). 

Indeed,  if  this  rule  is  satisfied,  then 

6  <  (a  —  l)(a(ti*)rxfc  —  b(uk)) 

<  (a  —  1)  inf{a(u)Txfc  —  6(u)  :  u  £  U), 

and  therefore, 

a(uk)r  xk  —  b(uk)  <  inf{a(u)Txfc  —  b(u)  :  u  £  U)  +  6 
<  a  ■  inf{a(u)rxi  —  b(u)  :  u  £  U}. 

(4)  The  most  violated  cut  a(u*)Txk—b{u*)  >  0,  where  a(u*)Txk—b(u*)  =  inf{a(tt)rxfc—  | 
b(u)  :  u  £  [/},  may  not  exist.  Even  if  it  does  exist,  it  could  be  in  general  very  hard  to  find. 
The  parameter  a  specified  in  Step  1  controls  the  strength  of  the  cut  generated  in  Step  3. 
When  or  is  close  to  one,  the  cut  is  near  the  most  violated  cut  when  it  exists.  When  a  is 
sufficiently  close  to  zero,  the  cut  is  very  weak,  nevertheless  the  algorithm  will  still  converge. 

It  is  easy  to  see  that,  if  Algorithm  4.1  stops  at  a  certain  iteration  k,  then  it  either  finds 
an  optimal  solution  of  (SIL)  or  detects  the  infeasibility  of  (SIL).  We  are  now  going  to  prove 
that  if  the  algorithm  does  not  stop  finitely,  then  any  cluster  point  of  the  sequence  xk  for  all 
Jfc  =  1,  2,  . . .  generated  in  Step  2  is  an  optimal  solution  of  (SIL). 

Definition  4.1  (e-optimal  solution).  Let  e  >  0.  A  vector  x  is  an  c-optimal  solution  of  (SIL) 
if  a(u)Tx  —  b(u)  >  —(  for  all  u  £  U  and  cTx  <  u(SIL),  where  t>(SIL)  denotes  the  optimal 
objective  function  value  of  (SIL). 

Theorem  4.1.  Suppose  that  {||(a(u), 6(xt))||00  :  u  £  U}  is  bounded  by  K  >  0  and  that 
the  algorithm  does  not  stop  finitely.  Let  the  sequence  x\  x2 ,  ...  be  generated  by  Step  2  of 
Algorithm  4.1.  Then, 
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(a)  (Finite  e-convergence)  For  any  e  satisfying  0  <  e  <  a,  there  exists  an  integer  N(e) 
such  that  xk  is  an  e-optimal  solution  of  (SIL)  for  all  k  >  N(c). 

(b)  (Convergence)  Any  cluster  point  of  the  sequence  x1 ,  x2,  . . .  is  an  optimal  solution 
of  (SIL). 

Proof.  Let  W  =  {s  £  fln+1  :  ||s||oo  <  A'}  and  G(x,y,z)  =  yTx  —  z,  where  x,  y  £  Rn  and 
z  £  R1 .  Then,  G(x,y,  z)  is  uniformly  continuous  on  T  x  W.  If  (a)  does  not  hold,  then  for 
some  0  <  e'  <  a,  N(e')  does  not  exist.  Therefore,  the  set  J  =  {Jfc  :  xk  is  not  an  e'-optimal 
solution  of  (SIL)}  contains  infinitely  many  integers.  Let  q  =  are'  >  0.  By  the  uniform 
continuity  of  G(x,y,z),  there  exists  6(q)  >  0  such  that 


||(z,  y,  z)  -  ( x ,  y,  £)||  <  6(q)  implies  that  |G(x,  y,  z)  -  G(x,  y,  z)|  <  q 


(4.2) 


for  all  (x,  y,  z)  and  (x,y,z)  £  T  x  W. 

Since  {(a(u),  6(u))  :  u  £  U]  is  bounded  and  J  is  infinite,  we  can  find  i,  j  £  J  with  i  <  j 
such  that 


||(x>,a(«-),i(ui))  -  (*>,a(u>),6(u*))||  =  ||(a(u*),  6(u'))  -  (a(u>),  &(«>)) || 


(4.3) 


Because  i  <  j,  we  have 


G(xJ,  a(u'),  6(u‘))  =  a(u*)rxJ  —  b(u’)  >  0.  (4-4) 

Because  j  £  J  and  cTxJ  <  u(SIL),  there  exists  u  £  U  such  that  a(u)r X*  —  6(u)  <  — e'  and, 
thus,  inf {a(u)TxJ  —  b(u)  :  u  £  U}  <  —e1.  Therefore,  by  Step  3  of  the  algorithm  we  have 
either 

G(xJ ,  a(uJ),  b(uJ))  =  a(u^)Tx}  —  b(u})  <  —a  <  —q  (4.5a) 

or 

G(xJ,  a(ttJ),b(u3))  =  a(u*)TzJ  —b(uJ) 

<  ainf{a(u)rx;  -  b(u)  :  u  £  U)  (4.56) 

<  -a-f'  =  -q. 

However,  (4.3),  (4.4),  (4.5a),  and  (4.5b)  contradict  (4.2).  Consequently,  (a)  is  proved. 
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It  remains  to  prove  (b).  Since  the  sequence  xk,  k  =  1,  2,  ...  is  bounded,  it  has  at  least 
one  cluster  point.  Let  x*  be  a  cluster  point  of  this  sequence.  From  (a)  we  know  that  for 
any  e  satisfying  0  <  e  <  a  and  k  >  N(e)  we  have  a(u)T*fc  —  6(u)  >  -e  for  all  u  £  U .  Hence, 
for  any  e  satisfying  0  <  e  <  a,  we  have  a(u)T x*  —  b(u)  >  —e  for  all  u  £  U,  which  implies 
that  i*  is  a  feasible  solution  of  (SIL).  To  complete  the  proof,  we  need  only  to  show  that 
cTx*  =  u(SIL).  First, we  know  that  cTx *  >  v(SIL)  since  x*  is  a  feasible  solution  of  (SIL). 
Secondly,  we  know  that  cTxk  <  u(SIL)  holds  for  all  k  since  the  feasible  region  of  LP(k) 
contains  that  of  (SIL).  It  follows  that  cTx*  =  u(SIL).  | 

Remark  4.1. 

(1)  We  have  proved  that,  if  the  algorithm  goes  on  infinitely,  then  any  cluster  point 
of  the  sequence  x1 ,  x2,  . . .  is  a  feasible  solution  of  (SIL).  Therefore,  if  (SIL)  is  infeasible, 
then  the  algorithm  will  detect  infeasibility  after  a  finite  number  of  major  iterations  (i.e., 
Step  2-Step  3). 

(2)  Without  loss  of  generality,  we  may  always  assume  that  {(|(a(u),  6(u))||00  :  u  £  U}  is 
bounded.  For  we  can  pre-multiply  (a(u),b(u))  by  a  positive  number,  e.g.,  ||(a(ti),  6(u))||_1 
and  the  feasible  region  remains  the  same. 

(3)  We  proved  the  convergence  of  Algorithm  4.1  without  assuming  the  continuity  of 
a(u)  and  b(u),  and  without  assuming  the  compactness  of  U .  We  are  able  to  do  so  because 
G(x,  y,  z)  =  yTx  —  z  is  uniformly  continuous  on  T  x  W  ■ 

In  the  rest  of  this  section,  we  discuss  how  to  solve  a  semi-infinite  linear  program  that 
does  not  have  the  constraint  x  £  T. 

Suppose  we  know  that  for  u1 £  U  the  linear  program 
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LP(0)'  minimize  crx 
subject  to 

a(u‘)rz  >  6(u*)  for  i  =  1,  . . t 

has  an  optimal  solution.  Then  we  can  start  the  algorithm  with  LP(0)'  as  LP(0).  Since  the 
feasible  region  of  LP(Jb)  contains  that  of  LP(ifc-f  1),  either  LP(fc  +  l)  is  infeasible  or  LP(fc+l) 
is  optimal.  Hence,  Algorithm  4.1  can  be  carried  out.  If  the  algorithm  stops  finitely,  then  it 
correctly  solves  the  problem.  Otherwise,  a  sequence  x1 ,  z2,  ...  will  be  generated.  It  is  easy 
to  see  that,  if  this  sequence  is  bounded,  then  both  (a)  and  (b)  of  Theorem  4.1  hold.  If  this 
sequence  is  not  bounded  but  has  a  cluster  point,  then  (b)  of  Theorem  4.1  still  holds  and 
(a)  should  be  stated  as 

(a)'  For  any  0  <  e  <  a,  there  exists  an  x‘  in  this  sequence  such  that  z*  is  an  e-optimal 
solution  of  (SIL). 

A  sufficient  condition  that  guarantees  the  boundedness  of  the  sequence  x1 ,  x2,  . . .  was 
given  by  Dantzig  (see  Dantzig  (1963)  pp.476-477)  for  the  Dantzig- Wolfe  algorithm.  This 
condition  works  for  our  algorithm  after  some  minor  modifications. 

Lemma  4.1.  Let  DP(fc)  be  the  dual  program  of  LP(fc)  for  all  k.  If  (SIL)  is  feasible  and 
a  nondegenerate  basic  feasible  solution  exists  for  some  DP(k),  then  the  sequence  xk  is 
bounded. 

Proof.  As  in  ordinary  linear  programming,  if  (SIL)  is  feasible,  then  v(SID)  <  +oo.  Let 
Bk  be  the  basis  associated  with  a  nondegenerate  basic  feasible  solution  of  some  DP(lfc) 
and  bk  be  the  vector  of  corresponding  cost  coefficients.  Then,  by  nondegeneracy,  we  have 
(B*)-1c  >  0.  Let  7*  =  (x‘)rBk  -  ( bk)T  for  all  i  =  1,2,....  Since  z*  is  feasible  for  LP(t') 
and  for  all  i  >  k  the  feasible  region  of  LP(i)  contains 
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that  of  LP(fc),  we  know  that  7*  >  0  for  all  i  >  k.  Hence,  for  all  i  >  k,  we  have 

0<y'(Bk)~'c 
=  (x‘)Tc  —  (6fc)r(Bk)-,c 
=  n(DP(i))  -  (ifc)T(fl*)-,c 
<  u(SID)  —  (fc*)r(f?*)-1c. 

It  follows  that  7*  is  bounded  and  thus  x*  is  bounded.  | 

Other  simple  sufficient  conditions  for  the  boundedness  of  xk  or  for  the  existence  of  a 
cluster  point  of  xk  are: 

Lemma  4.2. 

(1)  If  a(uJ)  <  0  for  some  uJ  generated  by  the  algorithm  and  xk  >  0  for  all  sufficiently 
large  k,  then  the  sequence  xk  is  bounded. 

(2)  Ifa(uJ)  <  0  for  some  uJ  generated  by  the  algorithm  and  xk  >  0  for  infinitely  many 
k,  then  the  sequence  xk  has  a  cluster  point. 

(3)  The  assumption,  a(tiJ)  <  0  for  some  u}  generated  by  the  algorithm,  in  (1)  and 
(2)  may  be  replaced  by  :  there  exist  uJ' ,  . . .,  uJl  generated  by  the  algorithm  such  that 
a(uJNi)  <  0  for  i  =  1,  . .. ,  l  and  a(u-»)  <  0. 

Proof.  (1).  For  all  xk  >  0  and  k  >  j,  we  have,  by  the  feasibility  of  xk ,  a(ti-?)rx*  >  b{u ■>). 
Therefore,  xk  <  6(uJ)/a(uJ  )i  for  all  i  =  1,  . . .,  n  and  thus  (1)  is  proved.  (2)  and  (3)  can  be 
proved  similiarly.  | 

4.3  A  Duality  Thoerem 

It  is  well  known  that,  if  a  primal  linear  program  has  an  optimal  solution,  then  its  dual 
program  also  has  an  optimal  solution  and  the  optimal  objective  function  values  of  the 
primal  program  and  the  dual  program  are  equal.  This  nice  property  does  not  hold  for  all 
semi-infinite  linear  programs  (see,  e.g.,  Duffin  and  Karlovitz  (1965)).  Herein  we  show  that 
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if  Algorithm  4.1  converges  for  a  primal  semi-infinite  linear  program,  then  there  is  no  duality 
gap  between  the  primal  program  and  its  dual  program  and  Algorithm  4.1  solves  the  primal 
program  and  the  dual  program  simultaneously. 

Let  DP(ik)  be  the  dual  program  of  LP(ifc)  for  all  Jfc,  where  LP(lfc)  is  generated  by  Al¬ 
gorithm  4.1,  and  y*  be  an  optimal  solution  of  DP (k).  If  at  a  certain  iteration  k ,  LP(fc) 
is  infeasible,  then  by  the  duality  theorem  of  linear  programming,  we  have  v(DP(fc))  =  oo. 
Since  the  feasible  region  of  DP(jfc)  is  a  subset  of  that  of  (SID),  we  know  that  in  this  case 
v(SID)  =  oo.  If  at  a  certain  iteration  k,  xk  is  an  optimal  solution  of  (SIL),  then  by  the 
duality  theory  of  linear  programming,  we  have  v(SIL)  =  cT  xk  =  u(LP(ifc))  =  v(DP(ib)).  On 
the  other  hand,  it  is  easy  to  show  that  u(SID)  <  t/(SIL).  Since  yk  is  a  feasible  solution  of 
(SID)  and  the  objective  function  value  of  (SID)  at  yk  is  equal  to  u(SIL),  it  follows  that  y* 
is  an  optimal  solution  of  (SID).  If  Algorithm  4.1  generates  an  infinite  sequence  xk  for  all 
k  =  1,  2,  . . .  with  a  cluster  point  x*,  then  we  have 

cTxk  =  v(LP(fc))  =  u(DP (*))  <  v(SID)  <  v(SIL).  (4.6) 

Since  the  feasible  region  of  LP(ib)  contains  that  of  LP(ik  +  1),  t>(LP(fc))  =  cTxk  is  nonde¬ 
creasing.  Hence,  we  have  limjt_00crx*  =  cTx*.  By  Theorem  4.1,  cTx*  =  v(SIL).  It  follows 
from  (4.6)  that  cTx*  =  v(SIL)  =  r(SID).  Now  suppose  that  the  method  used  for  solving 
LP(i)  in  Step  2  can  solve  LP (k)  and  DP (k)  simultaneously  (e.g.,  the  simplex  method)  and 
yk  is  an  optimal  solution  of  DP(fc)  generated  by  this  method.  Then,  y*  for  Jfc  =  1,  2,  . . .  is 
a  sequence  of  feasible  solutions  of  (SID)  on  which  the  objective  function  tends  to  v(SID). 
Thus,  the  following  theorem  is  established. 

Theorem  4.2.  Suppose  that  Algorithm  4.1  convergences. 

(1)  If  ( SIL)  is  infeasible,  then  v(SID)  =  oo. 

(2)  If  (SIL)  has  an  optimal  solution,  then  v(SIL)  =  v(SID). 

(3)  Algorithm  4.1  solves  (SIL)  and  (SID)  simultaneously.  | 


Corollary  4.1.  Suppose  that  {(a(u),  b(u))  :  u  €  (/}  is  compact,  the  feasible  region  of  (SIL) 
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has  an  interior  point,  v(SID)  is  finite,  and  Algorithm  1  converges.  Then,  v(SID)  can  be 
attained. 

Proof.  Let  x  be  an  interior  point  of  the  feasible  region  of  (SIL),  then  a(u)Tx  >  b(u)  for  all 
u  £  U.  As  {(a(u),6(u))  :  u  £  U]  is  compact,  we  know  that  there  exists  a  positive  number 
0  such  that  a(u)rx  —  6(u)  >  9  >  0  for  all  u  £  U.  Let  y*  be  the  optimal  solution  of  DP(ib) 
generated  by  Algorithm  4.1  and  A*  be  the  index  set  such  that  yk  >  0  if  and  only  if »  £  A*. 
If  we  solve  LP(fc)  and  DP(fc)  by  the  simplex  method,  then  A*  contains  no  more  than  n 
elements.  Let  |A*|  denote  the  cardinal  number  of  A*  and  Yk  =  (y*  , . . .  ,  yf  ,  0, . . .  ,  0)  € 

*i  *|a4| 

Rn,  that  is,  the  first  |A*|  components  of  Yk  are  y*^  , . . .  ,  yt*  ^  ^  and  the  rest  are  zero  in 

the  case  |A*|  <  n. 

We  claim  that  {T*  £  Rn  :  k  =  1, 2, . . .  }  is  bounded.  Indeed, 

9  Y  -  Y  a(«‘)Tiy*  -  Y  b(u')y? 

•€  A*  «€A» 

=  cTi  -  v(DP(k)) 

=  cTi-cTxk 
<  cT i  —  cT x* . 

Let  Ak  =  (a(u**' ), . . .  ,  a(u**iA»i ),  0, . . .  , 0)  £  Rn*n  and 

bk  =  (6(u**’ ),...,  6(u,*ia*i  ),  0, .  . .  ,0)T  £  Rn  for  all  Jb  =  1,  2,  -  Then,  we  have 

AkYk  =  c  and  ( bk)rYk  =  v(DP(k))  for  all  k  =  1,  2,  ....  Without  loss  of  gener¬ 
ality  we  assume  that  lim*_no  Ak  =  (o(u-'1 ),..., a(uJr), 0, ..,  0)  =  A*,  limjt_oofcfc  = 

(K«*) . 6(«-"),0,  -  -  -  ,0)  5  b\  and  lim  V*  =  (V,*,...  ,  V  * ,  0 - ,0)  =  Y\  where 

r  <  n.  Then,  A*Y*  =  c,  ( b*)rY *  =  v(SID),  and  thus,  y,,  =  V,*,  . . . ,  y,r  =  Y*,  y,  =  0  for 
all  i  £  {ji ,  •  •  •  ,  jr)  is  an  optimal  solution  of  (SID).  | 

4.4  Estimation  of  Upper  Bounds 

We  have  shown  that  for  any  e  satisfying  0  <  c  <  a,  the  algorithm  can  find  an  e-optimal 
solution  of  (SIL)  after  finitely  many  iterations  (Theorem  4.1).  Now  we  estimate  how  good 
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an  e-optimal  soiution  is  compared  to  an  optimal  solution.  In  this  section,  we  assume  that 
U  is  compact,  a(u)  and  b(u)  are  continuous  on  U ,  and  the  feasible  region  of  (SIL)  is  n- 
dimensional.  First,  we  state  two  lemmas  that  we  are  going  to  use  in  the  following  discussion. 

Lemma  4.3A  (Hoffman).  Let  A  be  an  (mxn)-matrix  and  b  an  m-vector.  If  Ax  >  b  has 
a  solution,  then  there  exists  a  constant  r  such  that  for  any  x  £  Rn,  there  exists  an  x° 
satisfying 

Ax°  >  b  and  ||x  -  x°\\  <  r\\(Ax  -  i>)~||oo, 

where  (Ax  —  b)~  stands  for  the  negative  part  of  the  vector  (Ax  —  b),  see  Hoffman  (1952).  | 

Lemma  4.3B.  If  { (a(ti),6(u))  :  u  £  U]  is  compact  and  F  =  [x  G  Rn  :  a(u)T x  >  6(u)  u  £ 
U}  is  n-dimensional,  then  a  given  halfspace  (a*)rx  >  6*  (where  a*  ^  0)  contains  F  if  and 
only  if  there  exist  u'  £  U  and  A,  >  0,  i  =  1,  . . . ,  t  (where  t  <  n)  such  that 

A,a(u1)  H - \-  A <a(u')  =  a* 

and 

A16(u1  )  +  ■••  +  A#6(tx< )  >  6*. 

(see,  e.g.,  Rockafellar  (1972),  p.160  )  | 

Definition  4.2.  Let  g  :  F  —*  Rn.  A  vector  x  £  F  is  a  stationary  point  of  the  pair  (F,  g)  if 
xr g(x)  <  yTg(x)  for  all  y  £  F. 

Suppose  that  xk  is  an  €-optimal  solution  of  (SIL).  Consider  an  auxiliary  program 

(P)  min{||x  —  **||2  :  a(tt)Tx  —  b(u)  >  0  for  all  u  £  U). 

Let  x *  be  the  optimal  solution  of  the  auxiliary  program  (P).  It  is  easy  to  see  that  x*  is 
a  stationary  point  of  (F,  V|jx  —  i*||2),  where  F  =  {x  £  Rn  :  a(u)Tx  >  b(u)  u  £  U)  and 
V||x  —  x*||2  =  2x  -  2x*  is  the  gradient  of  ||x  —  **j|2.  Therefore,  the  halfspace 

2(x*  -  xk)Tx  >  2(x*  -  xfc)rx* 
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contains  F.  It  follows  from  Lemma  4.3B  that  there  exist  u*  G  U  and  Aj  >  0,  i  =  1,  . . . ,  t 


(where  t  <  n)  such  that 


We  claim  that 


Aia(u’  )  +  •••  +  A<a(u*)  =  2(2*  —  xk) 

A,6(u’)  +  ••  +  A,6(u')  >  2(2*  -2*)T2*. 

a(ul)T  x*  =  6(u‘)  for  i  =  1, . . .  ,  t. 

i\T„*  _  L/„i\ 


(4.7a) 


(4.76) 


If  not,  say,  a(u')Tx*  >  b(ul)  for  *  =  1,  . . . ,  r,  and  a(u')T x*  =  6(u‘)  for  »  =  r  +  1,  . . . ,  t. 
Then,  we  have 

2(2*  -  xk)Tx*  =  Y1  ^a(u*)r2* 

*=1 

1 

>£>6K) 

«=i 

>  2(2*  -  2*)r2*, 

which  is  a  contradiction.  Thus  (4.7b)  is  proven.  Since  ||2  —  2fc||2  is  a  convex  function  of  x , 
(4.7a)  and  (4.7b)  imply  that  2*  is  an  optimal  solution  of  the  following  program: 

P (k)  min{||2  -  2*||2  :  Akx  >  6*}, 

where  Ak  =  (a(u’ ),..., a(u*))T  and  bk  =  (6(u’ ),...,  6(u#))r .  Applying  Hoffman’s  Lemma 
to  the  finite  system  of  linear  inequalities  Ak x  >  bk,  we  know  that  there  exists  a  constant  r 


and  an  x°  satisfying 


Akx°  >  bk  and  ||2*  -  2,,||  <  r||(A*2fc  -  bk)  ||<»  <  re. 


Because  2*  is  an  optimal  solution  of  P(fc)  and  xn  is  a  feasible  solution  of  P (fc),  we  have 


12*  -  2*11  <  ||2*  -  2^11  <  re 


and  therefore, 


|cr2*  -  cr2*|  <  re||c||  and  -  re[|c||  4-  cT 2*  <  cT xk  <  v(SIL)  <  cTx 


T  *  -  T  k 


Thus,  the  following  theorem  is  proved. 
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Theorem  4.3.  The  distance  between  an  e-optimal  solution  and  the  feasible  region  is  no 
greater  than  re,  and  the  difference  between  the  objective  function  value  of  an  (-optimal 
solution  and  the  optimal  objective  function  value  is  no  greater  than  7“e||c|| .  | 

There  is  a  simple  way  to  estimate  e  for  the  k-th  approximate  solution  xk  such  that  xk 
is  an  e-optimal  solution.  Suppose  that 

a(uk)r xk  —  b(uk)  <  a  •  inf  {a(u)r  a;*  —  b(u)  :  u  6  [/} 

of  Step  3  holds  for  xk  and  u*.  Let 

tn*  =  min{|(a(«*)rx*  -  6(u'))  -  ( a(uk)Txk  -  6(u*))|}. 

«<* 

Since  a(ti‘)rx*  —  &(«*)  >  0  for  all  i  <  k,  we  have  a(uk)rxk  —  b(uk)  >  —wk.  This  implies 
that  inf {a(u)T  xk  —  6(u)  :«€(/}>  —  wka~' ,  i.e.,  xk  is  a  (tufcar~  ^-optimal  solution 

Finally,  we  give  an  upper  bound  of  the  total  number  of  iterations  needed  for  finding 
an  e-optimal  solution.  We  need  the  following  assumptions: 

(A5)  a(u)  and  6(u)  are  Lipschitz  continuous  on  U  with  Lipschitz  constants  L 2  and  Ln 
respectively,  i.e.,  ||a(u)  —  a(u)||  <  L2||u  —  ti||  and  |6(u)  -  6(u)|  <  £.i||u  —  u||  for  all  u,  u  €  U ■ 
(A6)  ||**U  <  K  for  all  k  —  1,  2,  . . . ,  where  xk  is  generated  by  Step  2  of  Algorithm  1. 

(A7)  Every  cut,  a(uk)Tx  —  b(uk)  >  0,  generated  by  Step  3  of  Algorithm  1  is  the  most 
violated  cut. 

Lemma  4.4.  If  assumptions  (A5)  and  (A6)  are  satisfied  and  a(u)rxk  —  b(u)  >  0,  then 
a(u)rxk  —  b(u)  >  —e  for  all  u  that  satisfies  ||u  —  u||  <  e(L<iK  4-  • 

Proof. 

|a(u)TV  -  b(u)  -  (a{u)rxk  -  6(u))|  <  |)a(u)  -  a(u)||||a:*||  +  |6(u)  -  b(u)\ 

<  (L2I<  +  liOHu  -  ujl 
<e  I 
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Let  Bm(u,r)  denote  the  m-dimensional  Euclidean  ball  with  center  u  £  Rm  and  radius 
r.  If  xk  is  not  an  e-optimai  solution,  then  a(uk)Txk  —  b(uk)  <  —e  (by  (A7)).  Since 
a(u')Txk  —  6(u*)  >  0  for  all  i  <  k,  we  have,  by  Lemma  4.4, 

k—  I 

nk  $  (J  Bm(u\e(L2K  +  (4.8) 

i=i 

On  the  other  hand,  since  U  C  Rm  is  compact,  we  can  find  a  box  Q  C  Rm  containing 
U .  Let  /?  be  the  side  length  of  Q  and  M  be  the  smallest  integer  that  is  greater  than 
y/m0/{e(L2K  +  La)-1),  i.e., 

M  =  [y/mPMLtK  +  La)-1)]  +  1. 

Let  us  divide  Q  into  Mm  subboxes  of  the  same  m-content,  Ql ,  t  =  1,  . . . ,  Mm ,  satisfying 
int(Q‘)  flint(QJ)  =  0  for  all  i  ^  j,  where  int(Q')  denotes  the  interior  of  Ql .  For  every  uk 
generated  by  the  algorithm,  we  can  find  a  subbox  Q'k  containing  uk .  Since  the  diameter  of 
Q,k  is  y/mf3/M  and  y/mfd/M  <  e{L2I\  -f  La)-1,  we  have 

Bm(uk,e(L2K  +  La)-1)  3  Q'k .  (4.9) 

From  (4.8)  and  (4.9)  we  know  that  uk  £  (J>=i  Qtf  anc*  Q'k  £  {Q*1  >  ■  ■  ■  >  Conse¬ 

quently,  the  following  theorem  is  established. 

Theorem  4.4.  If  assumptions  (A5),(A6),  and  (A7)  are  satisfied,  then  an  upper  bound  of 
the  number  of  major  iterations  needed  for  finding  an  e-optimal  solution  is:  ([^/m  (J/(e(L2K  -f 

L*)~')]+l)m.  I 

4.5  Nonlinear  Semi- Infinite  Programming 

We  now  show  that  Algorithm  4.1  can  be  generalized  to  solve  the  following  nonlinear  semi- 
infinie  program 
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(SIN)  minimize  f(x) 
subject  to 

g(x,  u)  >  0,  for  all  u  €  U 
x  €  S, 

where  U  and  5  are  nonempty  compact  convex  sets,  /( i)  is  continuous  on  S,  and  g(x,u)  is 
continuous  on  S  x  U  ■ 

Definition  4.3  (e-optimal  solution).  Let  e  >  0.  A  vector  x  is  an  e-optimal  solution  of 
(SIN)  if  g{x,u)  >  — e  for  all  u  £  U  and  f(x)  <  v(SIN),  where  u(SIN)  denotes  the  optimal 
objective  function  value  of  (SIN). 

Algorithm  4.2. 

Step  1  (Initialization). 

Let  k  0; 

let  a  be  a  constant  such  that  0  <  a  <  1; 
let  NP(0)  be  the  nonlinear  program 
minimize  f(x) 
subject  to 

x  e  5. 

Step  2. 

If  NP(ifc)  is  infeasible,  then  (SIN)  is  infeasible,  stop. 

Else,  find  an  optimal  solution  xk  of  NP(fc); 

if  g(xk,u)  >  0  for  all  u  £  U, 

then  xk  is  an  optimal  solution  of  (SIN),  stop. 

Step  3. 

Find  a  u*  €  U  such  that  g(xk,uk)  <  a'inin{j(i*,u)  :  u  €  U}\ 
form  NP(Jb  +  1)  by  adding  a  cut,  g(x ,  uk)  >  0,  to  NP(fc); 
k  :=  k  +  1; 
go  to  Step  2. 
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Notice  that  the  convergence  proof  of  Algorithm  4.1  is  based  on  the  uniform  continuity 
of  G(x,  y,  z)  on  T  x  W,  and  now  g(x,  u)  is  uniformly  continuous  on  S  x  U.  It  is  easy  to  see 
that  the  following  theorem  holds. 

Theorem  4.5.  Suppose  that  U  and  S  are  compact,  g(x,  u)  is  continuous  on  S  x  U,  and 
the  algorithm  does  not  stop  finitely.  Let  the  sequence  x1 ,  x2,  . . .  be  generated  by  Step  2  of 
Algorithm  4.2.  Then, 

(a)  (Finite  e-convergence)  For  any  e  satisfying  0  <  e  <  a,  there  exists  an  integer  Ar(e) 
such  that  xk  is  an  e-optimal  solution  of  (SIN)  for  all  k  >  N(e). 

(b)  (Convergence)  Any  cluster  point  of  the  sequence  x 1 ,  x2,  ...  is  an  optimal  solution 
of  (SIN).  | 

Remark  4.2.  If  NP(ifc)  is  a  difficult  nonlinear  program  to  solve,  one  might  want  to  find 
instead  an  x  such  that  g(x,u)  >  —  e  for  all  u  6  U  and  f(x)  <  v(SIN)  +  6,  where  6  >  0.  In 
this  case,  the  optimal  solution  xk  of  NP(ifc)  generated  in  Step  2  of  Algorithm  4.2  may  be 
replaced  by  any  xk  that  is  feasible  for  NP(£)  and  that  satisfies  f(xk)  <  v(NP(k))  -f  6. 
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5.1  Introduction 

The  idea  of  solving  an  ordinary  convex  program  by  semi-infinite  linear  programming  comes 
from  Dantzig  (1963)  Chapter  24.  There  he  showed  how  to  solve  an  ordinary  convex  program 
by  generalized  linear  programming.  His  algorithm  starts  with  a  feasible  solution  of  the 
convex  program  that  provides  a  nondegenerate  basic  feasible  solution  for  an  initial  restricted 
master  program  and  then  generates  and  solves  a  sequence  of  linear  programs.  As  we  have 
seen  in  Chapter  4,  a  generalized  linear  program  can  be  considered  as  a  semi-infinite  linear 
program  of  the  dual  type.  We  now  show  how  to  apply  Algorithm  4.1  to  a  certain  semi¬ 
infinite  linear  program  so  as  to  obtain  a  feasible  solution  of  the  convex  program.  Using  this 
feasible  solution  as  a  starting  point,  we  then  apply  Algorithm  4.1  to  another  semi-infinite 
linear  program  and  obtain  an  optimal  solution  of  the  convex  program.  In  particular,  for  a 
strongly  consistent  convex  program  Algorithm  4.1  can  find  a  feasible  solution  after  a  finite 
number  of  iterations. 
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5.2  Feasibility 

We  want  to  find  a  feasible  solution  of  the  convex  program 

(CP)  minimize  f(x) 

subject  to 

gi(x)  <  0  for  i  =  1,  . . . ,  m 
x€T, 

where  T  C  Rn  is  a  polytope,  and  f(x)  and  ffi(x)  for  i  =  1,  . . . ,  m  are  real  valued  convex 
functions  on  Rn . 

First,  we  construct  a  semi-infinite  linear  program  whose  optimal  solutions  can  provide 
feasible  solutions  to  (CP). 

Let  G(x)  =  (1,  —  </i(x), . . . ,  —gm(x))T  and  e*  be  the  i-th  unit  vector  in  i?m+1 .  Consider 
the  following  dual  pair  of  semi-infinite  linear  programs: 

(SL)  maximize  e,Ty 

subject  to 

G(x)Ty  <  0  for  a)\x£T 
0  <  y<  <  1  for  i  =  2,  . . . ,  m  +  1, 

and  for  all  finite  subset  {x1  :  i  €  A)  of  T 

(SD)  minimize  + - 1-  sm 

subject  to 

E,e*  G(x-)A,  -  E’.:,  e*+V.  +  E;ii  e*+1S.  =  e1 

Xi  >  0  for  all  i  £  A 

p,,  s,  >  0  for  i  =  1,  . . . ,  m. 

Lemma  5.1.  Algorithm  4.1  converges  for  (SL)  and  (SD),  and  t>(SL)  =  v(SD). 

Proof.  Let  us  pick  any  x°  £  T  and  start  Algorithm  4.1  with  the  linear  program 
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T 

LP(0)  maximize  e1  y 

subject  to 

G(*°)Ty  <  0 

0  <  y>  <  1  for  i  =  2,  . . . ,  m  +  1. 

It  is  easy  to  see  that  LP(Q)  has  an  optimal  solution.  Then,  Algorithm  4.1  generates  a 
sequence  of  linear  programs 

er » 

LP(Jfe)  maximize  e1  y 

subject  to 

G(xi)Ty  <  0  for  «  =  0,  1 . A* 

0  <  !/«  <  1  for  t  =  2,  . . . ,  m  -f  1, 

where  x*  €  T  for  i  =  0,  I,  . . . ,  k.  The  dual  of  LP(t)  is 

DP(fc)  minimize  s,  H - h  sm 

subject  to 

Ef-n  <H**)*i  -  +  Eil  =  e1 

A,  >  0  for  »  =  0,  1,  . . . ,  k 
M»,  s«  >  0  for  i  =  1,  . . . ,  m. 

Let  yk  be  the  optimal  solution  of  LP(lfc)  and  (A k,fik,sk)  be  the  optimal  solution  of 
DP(fc)  generated  by  Algorithm  4.1.  To  show  that  Algorithm  4.1  converges  for  (SL)  and 
(SD),  we  only  need  to  show  that  yk  for  k  =  0,  1,  2,  ...  is  bounded.  Indeed,  by  the  duality 
theorey  of  linear  programming,  we  have 

yk  =  e’ V  =  +  •••  +  **>  0. 

On  the  other  hand,  since  yk  is  a  feasible  solution  of  LP(k),  we  have 

m 

y*  <  Sh(*0)y2  +  •  •  •  +  yn,(a:n)ym+1  <  ]T]  |y,(x°)|. 

i=i 

Hence,  yk  for  k  —  0,  1,  2,  . . .  is  bounded.  Let  y*  be  a  cluster  point  of  the  sequence  y*. 
Then,  by  Theorems  4.1  and  4.2,  we  know  that  Algorithm  4.1  converges  for  (SL)  and  (SD) 
and  v(SL)  =  v(SD).  | 
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Lemma  5.2.  (CP)  is  feasible  if  and  only  if  v( SD)  =  0. 

Proof.  If  (CP)  is  feasible,  i.e.,  there  exists  an  i’  satisfying  gi(x')  <  0  for  *  =  1,  ... , 
m,  let  A  =  {1},  A]  =  1,  p’  =  —g^fx1 )  >  0  for  i  =  1,  . . . ,  m,  and  s’  =  0  for  »  =  1,  . . . ,  m. 
Then,  we  have 

tn  m  m 

<?(*’)A1  +  ^e,+1s’  =e’  and  £  s’  =0. 

i=i  i=i  t=i 

Since  v(SD)  >  0  and  we  have  a  feasible  solution  of  (SD)  whose  objective  function  value 
attains  zero,  we  know  that  u(SD)  =  0.  Conversely,  if  v(SD)  =  0,  then  by  Lemma  51  we 


lim  u(DP(Jb))  =  v(SD)  =  v(SL)  =  0. 


First,  let  us  consider  the  case  u(DP(fc))  =  0  for  some  k.  Let  (A t,/it,sfc)  be  the  optimal 
solution  of  DP(fc)  generated  by  Algorithm  4.1  and  xk  =  By  khe  feasibility  of 

Afc,  we  have  A*  >  0  and  ]T)*_n  A*  =  1 ,  and  thus  xk  £  T.  By  the  convexity  of  gj{x)  and  the 


feasibility  of  (Afc,  p*,  sk),  we  have  (for  all  j  =z  1,  . . . ,  m) 


i=0 


Namely,  xk  is  a  feasible  solution  of  (CP).  Now  let  us  consider  the  case  u(DP(Jk))  >  0  for  all  k 
and  limt_.no  v(DP(/fc))  =  0.  Let  (A *,pfc,  sk )  be  the  optimal  solution  of  DP(fc)  generated  by 

Algorithm  4.1,  and  xk  =  \kxl  £  T  for  all  k  =  0,  1, - Since  sk  <  s”  and 

s*  >  0  for  all  k,  the  sequence  sk  is  bounded.  Therefore,  the  sequence  p*  =  Yli=o  (?(**)^f+sk 
is  bounded.  As  T  is  convex  and  compact,  we  know  that  the  sequence  xk  has  a  cluster  point 
x*.  Without  loss  of  generality,  we  assume  that  xk  — *•  x* ,  p*  — ►  p*,  and  s*  -»  t*.  Then,  by 
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the  convexity  and  continuity  of  gj(x)  and  the  feasibility  of  (Afc,  pk,  sk),  we  have 


9j (*’)  =  9i( Jim  xk) 

k-~>nc 

'^k\ 


=  Jim  9}{x  ) 

K—+00 

k 

<  Jim  ^2  Xi9j(*') 

i= 0 

=  Jim  (-«j  +  sk) 

fc— *  OO 


<  o. 

Hence,  a;*  is  a  feasible  solution  of  (CP).  | 


So  far  we  have  shown  how  to  find  a  feasible  solution  of  (CP)  by  applying  Algorithm  4.1 
to  (SL)  and  (SD).  If  (CP)  has  a  feasible  solution,  then  in  the  case  t>(DP(ifc))  =  0  for  some  Jfc, 
xk  is  a  feasible  solution  of  (CP)  and  Algorithm  4.1  stops  finitely;  otherwise,  u(DP(fc))  — ►  0  as 
k  — *  oo  and  hence,  for  any  t  >  0,  there  exists  a  number  N(e)  >  0  such  that  0  <  v(DP(ifc))  <  e 
for  all  k  >  N(e).  This  implies  that  for  all  k  >  N(e),  we  have 

9j(*k)  <  Yl  Xi9j(xi)  =  ~t*j  +  sj  <  f- 

i=0 

That  is,  for  any  e  >  0  Algorithm  4.1  can  find  an  x  that  satisfies  gj(i :)  <  e  for  all  j  =  1,  . . . , 
m  after  a  finite  number  of  iterations.  If  we  know  that  there  exists  a  feasible  solution  x°  of 
(CP)  and  a  small  positive  number  6  such  that  g}(x°)  <  — <5  <  0  for  all  j  =  1,  . . . ,  m,  then 
let  §j{x)  =  <7j(x)  4-  6.  As  we  have  just  shown,  for  e  =  6/2  >  0,  Algorithm  4.1  can  find  an 
x  that  satisfies  gj(x)  <  e  after  finitely  many  iterations.  Since  gj{x)  =  gj(x)  +  we  have 
9jix)  ^  —5/2  <  0  for  all  j  =  1 . m.  Thus,  the  following  lemma  is  established. 

Lemma  5.3.  If  gj(x)  <  6  for  all  j  =  1,  . . . ,  m  has  solutions,  where  6  is  a  small  positive 
number,  then  Algorithm  4.1  can  find  a  feasible  solution  of  (CP)  after  a  finite  number  of 
iterations. 

5.3  Optimality 

We  have  seen  how  to  find  a  feasible  solution  of  a  convex  program  (CP)  by  solving  a  semi¬ 
infinite  linear  progeram  (SD).  Now  we  show  how  to  find  an  optimal  solution  of  (CP).  Let 
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G(x)  =  (l,yi(ar), . . .  ,gm(x))T  and  e*  be  the  i-th  unit  vector  in  Consider  the  dual 

pair  of  semi-infinite  linear  programs: 

(CSL)  maximize  e1  y 
subject  to 

G{x)T y  <  f(x)  for  all  x  6  T 
y*  <  0  for  *  =  2,  . . . ,  m  -f  1, 

and  for  all  finite  subset  { x *  :  i  €  A}  of  T 

(CSD)  minimize  £2»€A  f(x')X , 
subject  to 

Hie  a  G(**)A<  +  e’+Vi  =  e1 

\i>  0  for  all  i  6  A 
Pi  >  0  for  i  =  1,  . . . ,  m. 

Lemma  5.4.  If  ( CP)  is  strongly  consistent,  then  Algorithm  4.1  converges  for  (CSL)  and 
(CSD),  and  n(CSL)  =  v(CSD). 

Proof.  Find  an  r°  satisfying  <7i(xn)  <  0  for  t  =  1,  . . .  m  and  start  Algorithm  4.1  with  the 
linear  program 

LP(0)  maximize  e,Ty 
subject  to 

G(xa)Ty  <  f(xn) 

Vi  <  0  for  i  =  2,  . . . ,  m  +  1. 

It  is  easy  to  see  that  LP(0)  has  an  optimal  solution.  Then,  Algorithm  4.1  generates  a 
sequence  of  linear  programs 

LP(i)  maximize  e,Ty 
subject  to 

G(x,)Ty  <  /(r*)  for  i  =  0,  1,  . . . ,  k 
y,  <  0  for  i  =  2,  . . . ,  m  +  1, 
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where  x‘  €  T  for  i  =  0,  1,  . . . ,  k.  The  dual  of  LP(ifc)  is 

DP(fc)  minimize  E*=o  /(a:*)A, 
subject  to 

Aj  >  0  for  t  =  0,  1,  . . . ,  k 
Pi  >  0  for  i  =  1,  . . . ,  m. 

Let  y*  be  the  optimal  solution  of  LP(fc)  and  (A k,pk,sk)  be  the  optimal  solution  of 
DP(fc)  generated  by  Algorithm  4.1.  We  want  to  show  that  the  sequence  y*  is  bounded.  By 
the  optimality  of  yk  and  convexity  of  f{x),  we  have 

*  k 

y*  =  ,'V  =  J2  fix')**  <  ^2  ^  l/(*)|.  and 

1=0  i=n  1 

k  k 


Thus,  yf  is  bounded.  By  the  feasibility  of  yfc,  we  have 

yf  +  y,(*°)y2*  +  •  •  •  +  ffm(*°)y* +l  <  /(*«). 

Since  y,(x°)  <  0  for  all  »  =  1,  2 . m  and  y*  <  0  for  all  i  =  2,  . . . ,  m  -f  1,  we  have  (for 

all  k  =  0,  1,  ...) 

y,*+i  >  (/(*")  -  yi*)/s.(*°)  for  *  =  I, ...  ,  m. 

Thus  y*  is  bounded  and  Algorithm  4.1  converges  for  (CSL)  and  (CSD).  | 

Remark  5.1.  If  y*  is  an  optimal  solution  of  (CSL),  then  xk  =  Et=o  ^ix'  ‘a  an  optimal 
solution  of  (CP).  If  Algorithm  4.1  converges  in  the  limit,  then  any  cluster  point  of  the 
sequence  £J  =  Ei=o  ^ix'  's  an  optimal  of  (CP)  (see  Dantzig  (1963)  Chapter  24). 

Remark  5.2.  Suppose  that  (CP)  is  not  strongly  consistent  and  that  one  can  find  xl  £  T 
i  —  1,  ...  i  m  satisfying  gj(x')  <  0  for  all  i  =  1,  . . . ,  m  and  j  =  1,  . . . ,  m,  and  yi(x‘)  <  0  for 
all  i  =  1,  ... ,  m.  Then,  Algorithm  4.1  converges  for  (CSL)  and  (CSD)  (see  Lemma  4.2). 
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Semi-Infinite  programming,  that  allows  for  either  infinitely  many  constraints  or  infinitely  many 
variables  but  not  both,  is  a  natural  extension  of  ordinary  mathematical  programming.  There  are  many 
practical  as  well  as  theoretical  problems  in  which  the  constraints  depend  on  time  or  space  and  thus  can 
be  formulated  as  semi-infinite  programs.  The  focus  of  this  dissertation  is  on  formulating  and  solving 
semi-infinite  programming  problems.  The  main  results  include: 

(1)  An  algorithm  for  solving  a  matrix  rescaling  problem  formulated  as  a  semi-infinite  linear  pro¬ 
gram.  Sufficient  conditions  that  guarantee  finite  termination  are  discussed  and  computational  results 
are  reported. 

(2)  An  algorithm  for  solving  a  matrix  estimation  problem  equivalent  to  a  semi-infinite  quadratic 
program.  For  a  specified  constant,  this  algorithm  will  find  an  approximate  solution  after  finitely  many 
iterations,  or  will  tend  to  an  optimal  solution  in  the  limit.  An  upper  bound  on  the  total  number  of 
iterations  needed  for  finding  an  approximate  solution  is  given.  Computational  results  are  reported. 

(3)  A  one-phase  algorithm  for  solving  a  large  class  of  semi-infinite  linear  programming  problems. 
This  algorithm  has  several  advantages:  it  handles  feasibility  and  optimality  together  and  can  detect 
infeasibility  after  a  finite  number  of  iterations;  it  has  very  weak  restrictions  on  the  constraints;  it 
allows  cuts  that  are  not  near  the  most  violated  cut;  and  it  solves  the  primal  and  the  dual  problems 
simultaneously.  Upper  bounds  for  finding  an  f-optimal  solution  and  for  the  distance  between  an  e- 
optimal  solution  and  an  optimal  solution  are  given. 

(4)  Applications  of  the  above  algorithm  to  convex  programming.  First,  a  certain  semi-infinite  linear 
program  is  solved  by  this  algorithm  so  as  to  obtain  a  feasible  solution  of  a  convex  program.  Then, 
another  semi-infinite  linear  program  is  solved  by  this  algorithm  so  as  to  obtain  an  optimal  solution 
of  the  convex  program.  In  particular,  it  is  shown  that  for  a  strongly  consistent  convex  program  this 
algorithm  can  find  a  feasible  solution  after  a  finite  number  of  iterations. 
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